Divide string into a sublist of sentence with the maximum number of character

Question

Context:

I have a long article (on Wordpress). And I'm using Google Translate API (v2) to translate this article. This article has about 300 000 characters.

However, due to the limit of Google Translate API.

• I can only translate a segment of text with a limit of 5000 characters.

So now, I need to split my long article into several segments of sentences. Then I send these small segments of sentences to Google Translate API

Problem:

I have a very long string. For example this string.

string = "This is the first sentence. This is the second sentence."

I want to partition the string into non-overlapping substrings (sentence) of length n (maxcharacter). There is a function StringPartition["string",n], but it doesn't count for the sentence. So it will cut the string.

string = "This is the first sentence. This is the second sentence. \
This is the third sentence."
maxcharacter = 40;
StringPartition[string, maxcharacter]

The result:

    {"This is the first sentence. This is the ", "second sentence. This \
is the third sente"}

I would like to:

• partition the long string into a sub-list of sentence
• the length of the sub-list does not exceed the maxcharacter count

Answer 1

Starting with your data:

string = "This is the first sentence. This is the second sentence. This is the third sentence."

There are two (at least :) ) ways to get the sentences:

sentences1 = TextCases[string, "Sentence"];
sentences2 = StringSplit[string, ". "];

Notice that these produce slightly different results.

You can use StringTake to limit how long each string is (it works if given a list of string, so no need to use Map):

StringTake[sentences1, maxcharacter]

One problem here is that if a string is less than maxcharacter, then StringTake will generate errors. You can fix this with UpTo:

StringTake[sentences1, UpTo[maxcharacter]]

Answer 2

Let's get some text from Charles Dicken's Oliver Twist.

Define:

strCD = "Although I am not disposed to maintain that the being born \
in a workhouse,is in itself the most fortunate and enviable \
circumstance that can possibly befall a human being,I do mean to say \
that in this particular instance,it was the best thing for Oliver \
Twist that could by possibility have occurred.The fact is,that there \
was considerable difficulty in inducing Oliver to take upon himself \
the office of respiration,--a troublesome practice,but one which \
custom has rendered necessary to our easy existence;and for some time \
he lay gasping on a little flock mattress,rather unequally poised \
between this world and the next:the balance being decidedly in favour \
of the latter.Now,if,during this brief period,Oliver had been \
surrounded by careful grandmothers,anxious aunts,experienced \
nurses,and doctors of profound wisdom,he would most inevitably and \
indubitably have been killed in no time.There being nobody \
by,however,but a pauper old woman,who was rendered rather misty by an \
unwonted allowance of beer;and a parish surgeon who did such matters \
by contract;Oliver and Nature fought out the point between them.The \
result was,that,after a few struggles,Oliver breathed,sneezed,and \
proceeded to advertise to the inmates of the workhouse the fact of a \
new burden having been imposed upon the parish,by setting up as loud \
a cry as could reasonably have been expected from a male infant who \
had not been possessed of that very useful appendage,a voice,for a \
much longer space of time than three minutes and a quarter. As Oliver \
gave this first proof of the free and proper action of his lungs, the \
patchwork coverlet which was carelessly flung over the iron bedstead, \
rustled; the pale face of a young woman was raised feebly from the \
pillow; and a faint voice imperfectly articulated the words, 'Let me \
see the child, and die. The surgeon had been sitting with his face \
turned towards the fire: giving the palms of his hands a warm and a \
rub alternately. As the young woman spoke, he rose, and advancing to \
the bed's head, said, with more kindness than might have been \
expected of him:
'Oh, you must not talk about dying yet.'
'Lor bless her dear heart, no!' interposed the nurse, hastily 

depositing in her pocket a green glass bottle, the contents of which 

she had been tasting in a corner with evident satisfaction.
'Lor bless her dear heart, when she has lived as long as I have, sir, 

and had thirteen children of her own, and all on 'em dead except two, 

and them in the wurkus with me, she'll know better than to take on in 

that way, bless her dear heart! Think what it is to be a mother, 

there's a dear young lamb do.'
Apparently this consolatory perspective of a mother's prospects 

failed in producing its due effect. The patient shook her head, and 

stretched out her hand towards the child.
The surgeon deposited it in her arms. She imprinted her cold white 

lips passionately on its forehead; passed her hands over her face; 

gazed wildly round; shuddered; fell back--and died. They chafed her 

breast, hands, and temples; but the blood had stopped forever. They 

talked of hope and comfort. They had been strangers too long.
'It's all over, Mrs. Thingummy!' said the surgeon at last.
'Ah, poor dear, so it is!' said the nurse, picking up the cork of the 

green bottle, which had fallen out on the pillow, as she stooped to 

take up the child. 'Poor dear!'
'You needn't mind sending up to me, if the child cries, nurse,' said 

the surgeon, putting on his gloves with great deliberation. 'It's 

very likely it will be troublesome. Give it a little gruel if it 

is.' He put on his hat, and, pausing by the bed-side on his way to 

the door, added, 'She was a good-looking girl, too; where did she 

come from?'
'She was brought here last night,' replied the old woman, 'by the 

overseer's order. She was found lying in the street. She had walked 

some distance, for her shoes were worn to pieces; but where she came 

from, or where she was going to, nobody knows.'";

Split into sentences:

slist = TextSentences[strCD]

These are the number of characters in the sentences.

StringLength /@ slist
{300, 376, 216, 222, 402, 309, 127, 130, 40, 190, 241, 63, 99, 73, 

37, 147, 78, 32, 33, 58, 156, 12, 128, 43, 173, 83, 34, 131}

Define: (a function that I used in a recent answer.

cSplit3[inList_List, f_Function] := Module[{rem = inList, i},
  Reap[
    While[rem != {},
     i = 1;
     While[i <= Length@rem && f@Take[rem, i],
      i++;
      ];
     i--;
     Sow@rem[[1 ;; i]];
     rem = Drop[rem, i];
     ]
    ][[2, 1]]
  ]

Usage: (Here I have chosen max count of 600)

(chunks = cSplit3[slist, Total@StringLength@# < 600 &]) // Column
StringLength /@ chunks

{{300}, {376, 216}, {222}, {402}, {309, 127, 130}, {40, 190, 241, 
  63}, {99, 73, 37, 147, 78, 32, 33, 58}, {156, 12, 128, 43, 173, 
  83}, {34, 131}}

---

Output: (* see chunks *)

---

Original answer

Manipulate[
 StringJoin @@@ (Partition[Characters@string, UpTo[n]]) // TableForm
 , {{n, 10}, 10, 50, 1}
 ]

Answer 3

Okay, reading your updated question, maybe this is what you want:

Starting with some sample data to test with:

string = ExampleData[{"Text", "DeclarationOfIndependence"}];
StringLength@string
(* 8936 *)

Get the sentences:

sentences = TextSentences[string];
(* Thanks @MarcoB *)

Now, the strategy is to use Split to group sentences together if they're sufficiently short. Then we can map StringJoin over the result. Split can take a test predicate which takes two adjacent members of the list. We can add some statefulness to this process. It's a bit wonky, and maybe someone has a more elegant idea, but here's what I've got at this point (I've tried to format it to make everything easy to inspect):

results =
  Module[
    {len = StringLength@sentences[[1]],
     threshold = 1000},
    StringJoin /@
      Split[
        sentences,
        If[len + StringLength@#2 < threshold,
           len += StringLength@#2; True,
           len = StringLength@#2; False] &]]

Now, there are some consequences to this. Let's look at the length of the results:

StringLength[results]
(* {407, 965, 967, 879, 803, 1115, 925, 990, 943, 908} *)

Some snippets could be longer than the threshold (because of very long sentences). I assumed that you'd want to preserve long snippets rather than drop them.