You jointly describe and annotate a short video in ONE response. Produce a grounded caption, entity mentions linked to the supplied marks, and every action span in that same final caption with its subject and objects. The instructions below define the two annotation layers of ONE output. Examples using CAPTION/GROUPS below are illustrative; your actual input is images and a MARKS listing. You create the caption and entity mentions yourself. You write a GROUNDED CAPTION for a short video: a natural description of what happens, plus a list of which caption phrases refer to which of the video's annotated objects. ## What you are given 1. CLEAN KEYFRAMES — frames of the video with no overlay, in time order, each stamped "f ". These are your source for what the video SHOWS: appearance, colour, clothing, what moves, what happens first and next. 2. MARK OVERLAYS — a few of the same keyframes with the annotated objects painted in and an ID badge on each. When there are many objects the overlay for one frame is split into several panels ("marks 2/4"); each object always appears in the same panel and always in the same colour. These tell you WHERE each ID is. 3. IDENTITY SHEETS — one cell per annotated object: a crop centred on it, taken from the frame where it is largest, everything outside its mask darkened, labelled "# ". These tell you WHICH object an ID is. When several objects share a category (four people, six monitors), the identity sheet is how you tell them apart — use it, do not guess from the category list. 4. MARKS — a text listing of the same objects: id, category, which frames it appears in, its peak size as a fraction of the frame, its average centre as (x, y) in [0,1] with (0,0) at the top-left, how far that centre travels across the clip (fraction of the frame diagonal), and which overlay panel it is in. The video is short — the duration is given below. Look at the keyframes in order and work out what actually happens over that time. ## What to produce `caption` — plain prose describing the video. - Present tense. Use 2 to 3 SHORT sentences, roughly 15 to 30 words total. Keep each sentence simple, with one main action or state. Do not pack several clauses into a long sentence with repeated "while", "as", "with", or semicolons. Use up to about 40 words only when a crowded scene cannot be covered clearly in fewer words; never write a frame-by-frame transcript. - Say what HAPPENS, in the order it happens: who does what to what, what moves, what changes. Later stages extract every action from this caption and localise it in time, so events, motion and their order are the substance of the caption; a static inventory of objects is not. - Describe the setting in a clause or two where it matters (indoors/outdoors, the surface, the room, the weather), not as a list. - Use NATURAL English words, not the coarse category vocabulary. The listing may say "screen or television", "bottle or cup", "other animal", "tub or bowl or pot" — write "monitor", "mug", "goat", "bowl". Add the distinguishing detail a reader needs to tell two similar objects apart ("the man in the red shirt", "the left-hand monitor"). - Use ACTIVE, NATURAL COREFERENCE across sentences. Do not repeatedly restart with the same full noun phrase. For each salient object or person introduced in an early sentence and still involved later, make at least one safe cross-sentence reference using a pronoun, possessive, or shared plural pronoun. When that entity is the grammatical subject of the next sentence, prefer the pronoun explicitly: "A man reaches toward a dog. He touches its back." "A car enters the road. It passes the house." "An excavator lifts its boom. It then folds the arm." A repeated phrase such as "the car" or "the excavator" is not the preferred form when "it" is clear. Other good patterns are: "A woman sits beside a man. They watch the road." Use the first sentence to introduce the entity, the next sentence to refer back to it, and a third sentence only when it adds a new action or state. Every reference still gets its own mention entry linked to the same mark ID(s). If several same-category objects make a singular pronoun ambiguous, introduce distinguishing roles first; then use "he", "she", or "it" for the currently active entity and retain a short distinguishing phrase for the other entity, for example: "A woman touches a dark horse beside a pale horse. It lowers its head, while the pale horse steps right." Never add a pronoun without a real visual antecedent just to satisfy this rule. - When 9 or more instances share one category, do not mechanically enumerate them. Use a truthful collective phrase such as "many people", "a group of people", "many cars", or "a row of monitors" and link it to all relevant mark IDs. Use "some" only when the phrase intentionally refers to a subset, not to the whole inventory. The exact object inventory remains authoritative even when the natural wording is collective. - The category in MARKS is an identity hint, not a phrase that must be copied. Use the visually correct everyday noun: for example, a mark labelled "bat" may be described as a hockey stick if that is what the image shows. Do not force a wrong noun merely to make the lexical category check agree. - Only describe what is VISIBLE in the frames. Do not infer a purpose, a profession, a name, an emotion, speech, or sound. If something is unclear, describe it more vaguely rather than guessing; if you cannot tell whether an event occurs, leave it out. - The MARKS listing is a hard annotation boundary for captionable entities. If an entity is visible in the clean frame but has no corresponding mark in MARKS, do not mention it in the caption and do not create a mention for it. For example, if a cow is visible but no cow mask is listed, omit the cow rather than writing "cow". Never add an unannotated object merely to make the scene sound more complete. - Describe only visible people, animals, and physical objects, together with their visible actions, processes, or states. Do NOT describe the filming camera or camera viewpoint as a scene event: omit camera moves, pans, zooms, cuts, tracking/dolly motion, and handheld wobble from both the caption and the action content. - If the only apparent change is camera motion over otherwise static entities, describe the visible scene without mentioning the camera. A caption with no actions in it is correct for such a clip. - This exclusion applies to the filming camera/viewpoint, not to a physically visible camera device that is itself an object in the scene and is visibly handled, moved, or dropped. - NEVER write an ID, a number in brackets, a "#", or any other marker into the caption. The caption must read as ordinary prose. IDs belong only in `mentions`. `mentions` — one entry per phrase in your caption that refers to an annotated object. - `text`: the phrase, copied EXACTLY as it appears in your caption (same characters, same case, no surrounding punctuation). It must be a substring of the caption. - `occurrence`: which occurrence of that exact string in the caption you mean, counting from the start of the caption, 1-based. "the man" appearing three times gives three entries with occurrence 1, 2, 3. - `mark_ids`: the ID(s) this phrase refers to. - `confidence`: high / medium / low — how sure you are that this phrase refers to THAT object. Use low when two same-category objects were hard to tell apart. ### Span rules — follow the Text Span Annotation Guidelines - Include the article, quantifier, count, demonstrative, possessive, and adjectives before the head noun: "the dog", "a red car", "several people", "another man". Articles are never dropped. - A span contains the head noun plus premodifiers only. Exclude post-head prepositional phrases, relative clauses, and spatial modifiers: in "the woman in black", annotate "the woman"; in "a man with a hat", annotate "a man" and "a hat" separately. - Do not include punctuation, conjunctions, or prepositions in a span. Do not create nested spans within one group: use "his phone", not both "his" and "phone" in the phone group. Cross-group overlap is allowed for shared anchors and possessives. - Pronouns such as "he", "she", "it", "they", and "them" are valid spans when they clearly refer to visible entities. Put them in every antecedent instance group they refer to; "them" may therefore be shared by the woman and the man it refers to. - If a count phrase has explicit selectors, share the count anchor across the selected instance groups: "Three people ... one ... another ... the third". If there are no selectors, do not invent instance-specific assignments from the count alone. - Possessive rules: "his" belongs to the possessor; "his phone" belongs to the phone. Body parts are separate entities under the annotation guidelines, but only annotate a body-part span when the inventory contains a corresponding body-part mask. Otherwise avoid introducing a body-part object that cannot be grounded. - Annotate every concrete entity that is actually mentioned, including stuff such as floor, ice, sky, grass, and water. Do not annotate abstract words such as "background" or "foreground", and do not mention masks, overlays, IDs, or frames. ### Which phrases to mark - Mark the NOUN PHRASE: determiner, adjectives and the head noun ("a man", "the red car", "the left-hand monitor", "two dogs"). Keep it short — 1 to 4 words is typical. - Do NOT swallow following prepositional phrases or relative clauses, because those contain other objects: in "the man on the bicycle", mark "the man" and "the bicycle" separately. - Mark EVERY mention of an object you have grounded, including repeats, pronouns and possessives: "A man ... He ... his hand" gives three mentions, all pointing at that man's ID. Coreference chains are the point; a pronoun with no mention entry is a gap. - Body parts belong to their person's ID ("his hand" -> that person's ID): the annotation covers whole people, never parts. - A phrase naming SEVERAL annotated objects at once ("three people", "the two dogs", "a row of monitors") gets ALL of their IDs in `mark_ids`. Only do this when the phrase really is about all of them. - Do NOT mark a phrase that refers to something with no ID. Extra scenery that is not annotated simply gets no mention entry — that is expected and fine. - Do not mention an unmarked entity in the caption in the first place. Every concrete entity named in the caption must have at least one corresponding MARKS ID. - Never link a phrase to an ID whose category it contradicts. If the identity sheet shows #7 is a chair, no phrase about a person may carry ID 7. If you cannot decide which of two IDs a phrase means, pick the more likely one and set confidence low. ### Which objects to cover - Describe every annotated FOREGROUND object that plays a visible part — people, animals, vehicles, the objects being handled. Aim to ground all of them. - Background regions (wall, floor, ground, sky, grass, road, water, ceiling, building) are annotated too. Mention one only when it genuinely belongs in the description: the surface someone acts on, the water someone swims in, the sky in a landscape shot. Do not list the rest to raise coverage — a caption that recites the background is a worse caption. - Small or fleeting objects you cannot describe honestly are better left ungrounded than grounded by guesswork. # ACTION ANNOTATION RULES (caption_actions v2) ## What counts as an action Any predicate asserting a visually meaningful physical ACTION, PROCESS, EVENT, or held bodily posture: - actions/events: runs, throws, opens, scratches, hits - postures and bodily states: sitting, standing, seated, lying, perched, squatting - processes: moving, floating, drifting, falling A word being grammatically a VERB does NOT by itself make it an action. The verb must semantically express an actual visible action/process/event in the caption. For example: "He tries to jump." -> "jump", NOT "tries" "He prepares to run." -> "run", NOT "prepares" "a falling ball" -> "falling" "a fallen ball" -> nothing for "fallen" `try`, `attempt`, `prepare`, `plan`, `intend`, `want`, and similar verbs that merely introduce an intended, attempted, or prepared action are not themselves action events for this tier. Extract the actual embedded action instead. However, when both verbs independently denote real physical actions, extract BOTH: "He runs to hit the desk." -> "runs", "hit" "She bends to pick up the bag." -> "bends", "pick up" Mere presence, visibility, static spatial arrangement, mental state, intention, or resulting state is NOT an action (see the exclusion list below). ## Camera/viewpoint exclusion The filming camera and camera viewpoint are out of scope. If the caption says that "the camera" or the viewpoint pans, zooms, moves, tracks, shakes, or cuts, do NOT return that predicate as an action and do not return the camera/viewpoint as a subject or object. Return actions [] if that is the only apparent motion. This rule does not exclude a physically visible camera device when it is an ordinary scene object that is actually handled, moved, or dropped. Process EVERY sentence. Actions in subordinate clauses, coordinated clauses and reduced relative clauses count too. Every distinct event gets its own entry. ## POS guidance Use Penn Treebank POS tags as an important guide when identifying action heads and particles. Main action-head candidates: VB, VBD, VBG, VBN, VBP, VBZ Important neighboring tags: RP = particle · IN = preposition / subordinating conjunction · TO = infinitival "to" RB/RBR/RBS = adverb · MD = modal POS is a guide, not the final decision. A VB* token must still pass the semantic action test above. An RP token is strong evidence for a phrasal particle, but words such as up, down, in, out, on and off can have different grammatical functions in different contexts. Use the syntactic function and meaning in the current sentence: - genuine phrasal particle -> may be part of the action span - ordinary direction -> exclude - preposition heading a PP -> exclude If you reason about dependencies, a particle relation such as `prt` / `compound:prt` is strong evidence that the word is a genuine phrasal particle. ## One event, many mentions Captions often re-describe the SAME action of the same participants — a restatement, an elaboration, a continuation. One event = ONE entry, whose `mentions` array lists EVERY expression of it, in caption order (each as its own exact substring): "A man is pushing a cart. He pushes it into the garage." -> ONE action with mentions ["is pushing", "pushes"] (subject "A man", objects ["a cart"]) "A dog chases a ball. The dog keeps chasing it until it stops." -> ONE action with mentions ["chases", "chasing"] ("keeps" stays out) Each mention is the verb group of that expression (aux included, per the span rule), or its complete light-verb + action-noun construction when that exception applies; control/aspectual verbs like "keeps" are never part of a mention. Return SEPARATE entries when the caption describes distinct events, even of the same kind by the same participant: - explicit repetition: "He waves. He waves again." -> two actions - a new occurrence after other events: "She jumps, lands, then jumps onto the box." -> two jump actions Most actions are mentioned once: then `mentions` has a single element. ## Span boundaries -- the core rule The action span is the VERB GROUP of the core lexical action: its auxiliary chain (the VB/VBD/VBG/VBN/VBP/VBZ forms of be, get and have that belong to the same verb group) plus the lexical action verb, plus any genuine PHRASAL-VERB PARTICLE. For LIGHT-VERB / ACTIVITY-NOUN constructions, also include the noun phrase that names the action, exercise, gesture, sport, or physical activity (see below). Modals (MD: can, could, must, will, would, may, might, shall, should) are NOT part of the span — they express possibility or obligation, not the action itself. Neither are control/aspectual/intention verbs (see below). The goal is NOT to extract the entire verb phrase. Exclude independently groundable entity participants from the action span. Those belong in subject/objects instead. Also exclude modals, control verbs, ordinary directions, adverbs, ordinary prepositions, prepositional phrases, infinitival "to", and clause-linking function words. Auxiliary do/does/did is excluded, in all inflected forms: "He did fall." -> "fall"; "He does not move." -> "move". This boundary rule alone does not assert that a negated event occurred. Lexical do in "does a flip" is different: it supports an action noun and stays. INCLUDE in the span: - the verb group: VB* auxiliaries + the core lexical action verb "A fish is eating." -> "is eating" "Children are clapping." -> "are clapping" "Honey is fed using a syringe." -> "is fed" "The door gets pushed." -> "gets pushed" "The door is being pushed." -> "is being pushed" "The ball was kicked." -> "was kicked" but never a modal (MD is not a VB* tag): "She can swim." -> "swim" - a genuine phrasal particle that belongs to the lexical verb "Person opens dishwasher and pulls out rack." -> "opens", "pulls out" "Woman puts on black helmet." -> "puts on" "He picks up the ball." -> "picks up" "She turns on the light." -> "turns on" ### Light verbs and nouns naming physical activities Keep the WHOLE minimal contiguous construction when a verb such as do, perform, execute, make, give, take, or play combines with a noun phrase that NAMES the visible action, movement, exercise, gesture, maneuver, sport, or activity. The noun is the semantic action, not a separate concrete entity acted upon. Judge by meaning in context; this verb list and the examples are not exhaustive. Apply the rule to all inflections and progressive/perfect auxiliary chains. "A person performs jumps." -> "performs jumps", type light_verb, objects [] "He does a flip." -> "does a flip", type light_verb, objects [] "They do squats." -> "do squats", type light_verb, objects [] "She did a somersault." -> "did a somersault", type light_verb, objects [] "He is doing three push-ups." -> "is doing three push-ups", type light_verb "He does sports." -> "does sports", type light_verb, objects [] "She has performed a backflip." -> "has performed a backflip", type light_verb "He executes a forward roll." -> "executes a forward roll", type light_verb "She makes a jump." -> "makes a jump", type light_verb "He gives a thumbs-up." -> "gives a thumbs-up", type light_verb, objects [] "She takes a step." -> "takes a step", type light_verb "Two people play table tennis." -> "play table tennis", type nonvisual_object "A person is playing golf." -> "is playing golf", type nonvisual_object The light-verb rule is about grammatical function, not one fixed surface form. Apply it to all ordinary tense, person/number, and auxiliary-chain variants of the same construction. Preserve the exact contiguous words that appear in the caption: "He does sports." -> "does sports" "They did sports." -> "did sports" "She is doing sports." -> "is doing sports" "He has done sports." -> "has done sports" "They have been doing push-ups." -> "have been doing push-ups" Do not strip do/does/did/doing/done when the form is the lexical light verb followed by an action/activity noun. This is different from auxiliary do in "He did fall", where only "fall" is the action span. Include determiners, possessives, numbers and modifiers INSIDE the action-noun phrase, because they are part of the exact contiguous expression: "does a flip", "performs two backward somersaults". Exclude trailing location, equipment, manner and time adjuncts: "does a flip on the mat" -> "does a flip". An intervening adverb may stay only when required for one contiguous substring: "does slowly a flip" -> "does slowly a flip" (ungrammatical, low confidence). Do not rewrite "does a somersault" as "somersaults". Do not return a separate action for the same included noun, or put that noun in objects/entity groups. Concrete recipients/equipment may still be objects: "gives a thumbs-up to a man" -> "gives a thumbs-up", objects ["a man"]. For coordinated action nouns under ONE shared verb, retain one contiguous phrase and do not create overlapping mentions: "does squats and lunges" -> "does squats and lunges". Independently expressed/repeated events stay separate. Contrast concrete entities; these nouns stay OUT of the span: "He hits the desk." -> "hits", objects ["the desk"] "She gives a ball to a child." -> "gives", objects ["a ball", "a child"] "He plays the piano." -> "plays", objects ["the piano"] "She takes a cup." -> "takes", objects ["a cup"] Mental/abstract nouns are not automatically physical actions: "makes a decision" or "has an idea" does not become a visible action just because it has a light verb. "performs uniform" has no clear physical activity noun in ordinary English: do not automatically label it a light_verb action. If the intended action is unclear, omit it; never silently replace it with "performs jumps" or another phrase. ### Lexicalized physical-event complements Some physical events are expressed by a lexical verb together with an abstract complement rather than a concrete entity. Keep the minimal complete predicate when the combination names the visible event. In particular, keep the full balance-loss expression, including an overt possessive when present: "The kid loses balance." -> "loses balance", objects [] "The kid loses its balance." -> "loses its balance", objects [] "The kid lost balance." -> "lost balance", objects [] "The kid is losing balance." -> "is losing balance", objects [] Do not reduce these to only "loses", "lost", or "is losing". The abstract complement "balance" is part of this lexicalized action expression, not a separately groundable entity object. If an event-noun phrase is pronominalized, do not invent the missing noun text. Only keep "does it" as a light-verb mention if "it" clearly refers to that same physical event (e.g. "does a flip, then does it again"); otherwise keep the verb or omit an unclear event. Each repeated occurrence remains a separate event. A phrasal particle is different from an ordinary directional word or preposition: "He picks up the ball." -> "picks up" "He picks the ball up." -> "picks" + separated particle "up" But: "He drives up the hill." -> "drives" "He walks to the left." -> "walks" "He looks at the dog." -> "looks" "He sits on the chair." -> "sits" The deciding question: does the neighboring word belong to the lexical phrasal verb itself, or does it merely express direction/location or introduce a prepositional phrase? Only the first case belongs in the action span. ### Directional modifiers Do NOT include directional modifiers merely because they distinguish where the motion goes: "The car is driving forward." -> "is driving" "He flips backwards." -> "flips" "She falls backwards." -> "falls" "The kid turns left." -> "turns" "A man glides forward." -> "glides" "A man falls down." -> "falls" Direction belongs outside the action span unless the word is genuinely a lexical phrasal particle. ### Prepositions and prepositional phrases Do NOT include an ordinary preposition in the action span, including a preposition selected by the verb. The preposition and the noun phrase it introduces stay outside the action span: "Man looks at a dog." -> "looks" objects ["a dog"] "Woman waits for a train." -> "waits" objects ["a train"] "A person points at a laptop." -> "points" objects ["a laptop"] "Man walks in the park." -> "walks" "He is sitting on a chair." -> "is sitting" "A boy sits next to a woman." -> "sits" "He looks to the left." -> "looks" Do NOT keep a dangling preposition: "looks at" -> wrong · "sits on" -> wrong · "walks to" -> wrong Compare a true particle: "turns on the light" -> "turns on" with a preposition: "sits on the chair" -> "sits" ### Infinitival clauses Infinitival `to` is not part of an action span. If an intention/attempt/preparation verb introduces a genuine action, extract the embedded action only: "Man prepares to throw discus." -> "throw" "Man tries to jump." -> "jump" "Helicopter attempts to take off." -> "take off" Do NOT return "prepares to throw", "tries to jump", "attempts to take off". But if both predicates are independently real actions, return both separately: "He runs to hit the desk." -> "runs", "hit" "He steps forward to grab the bag." -> "steps", "grab" ### Participles VBG and VBN must be interpreted semantically, not mechanically. An ongoing visible process expressed by VBG counts: "a falling ball" -> "falling" "a moving car" -> "moving" A VBN that merely describes a completed/resulting state does not count: "a fallen ball" -> nothing for "fallen" "a broken chair" -> nothing for "broken" if it only describes the chair's state But an eventive passive DOES count, and keeps its auxiliary chain: "The ball was kicked." -> "was kicked" "The door is being pushed." -> "is being pushed" A bare participle with no auxiliary in the text stays bare: "a stick tied to a rope" -> "tied" The question is whether the caption asserts an action/event, rather than merely describing a resulting state. ## EXCLUDE entirely — these are NOT actions, return no entry for them: - STATIVE COPULAR RELATIONS: a copula plus a location, orientation, visibility or existence complement describes a configuration of the scene, not an action. "the two men are next to the black car." -> nothing "Two grey cars are in front." -> nothing "A woman in white is at a basin." -> nothing "A kid is on a balance bike." -> nothing "A white table is visible." -> nothing "a moving car is present." -> nothing for "is present" ("moving" is still an action) "A ping-pong table is in view." -> nothing "The bike is backward." / "is upside down." -> nothing "There are four pillows in front of it." -> nothing Contrast progressives and postures, which DO count with their auxiliary chain ("is walking", "is sitting", "standing"), and eventive passives, which are real events ("was thrown", "is fed"). The test: copula + VB* participle = a verb group = action; copula + location/adjective/existence = configuration = nothing. - intention, attempt and preparation predicates when they merely frame another action "He tries to jump." -> no action for "tries" "He prepares to throw." -> no action for "prepares" "She plans to run." -> no action for "plans" - result-state participles that do not assert a visible event "a fallen ball" -> no action for "fallen" "a broken chair" -> no action for "broken" when merely descriptive ## EXCLUDE from the span: - modals (MD) — but keep the VB* auxiliary chain "He is walking." -> "is walking" "She can swim." -> "swim" (modal out) "He must go now." -> "go" (modal out) - control / aspectual verbs that frame the core action "The dog keeps chasing it." -> "chasing" "He starts running." -> "running" "She continues to spin." -> "spin" This excludes aspectual start/stop only when it frames an embedded action: "She stops walking." -> "walking" Lexical start/stop acting on a concrete entity stays: "He starts the engine." -> "starts", objects ["the engine"] "She stops the car." -> "stops", objects ["the car"] Standalone start/stop stays if it clearly describes a physical event. - subordinating conjunctions (while, as, before, after, when) "A parrot spreads its wings while standing on the edge of a bowl." -> "spreads", "standing" NOT "while standing" - temporal and sequencing adverbs (then, finally, suddenly, at the same time, again) "He finally lands." -> "lands" "They squeeze the balloon at the same time." -> "squeeze" - manner, degree and ordinary directional adverbs "He walks angrily." -> "walks" "The car moves forward." -> "moves" "He hits him badly." -> "hits" - reciprocal pronouns (each other, one another) "Two high-speed trains pass each other." -> "pass" - objects that are CONCRETE VISIBLE THINGS; they go in the `objects` field instead "A bald man in a suit plays the piano." -> "plays" "A cat scratches a rooster." -> "scratches" - nouns and pronouns naming independently groundable entity participants; EXCEPTION: nouns naming the action/activity in the light-verb rule stay in the action span, and do not become concrete objects "He hits the desk." -> "hits" "He looks at me." -> "looks" "He picks up the ball." -> "picks up" "Two people playing table tennis." -> "playing table tennis" objects [] "A person performs jumps." -> "performs jumps" objects [] - ordinary prepositions and their noun phrases "Man looks at a dog." -> "looks" "He is sitting on a chair." -> "is sitting" "Man walks in the park." -> "walks" "Two men squatting on the ground." -> "squatting" - subjects, adverbials, and nominal events introduced by "during" "A girl grabs a red ball during a color sorting activity." -> "grabs" The test for the ACTION SPAN is: 1. Find the lexical verb that expresses the actual visible action. 2. Reject grammatical verbs that only express intention, attempt, preparation, planning, modality, or auxiliary information. 3. Include a genuine lexical phrasal particle; for light-verb/activity-noun constructions, include the minimal full action-noun phrase as well. 4. Exclude ordinary directions, prepositions, PPs, adverbs and entity nouns/pronouns, except internal modifiers/function words needed by the light-verb construction. ## Separated particles Every span you return must be ONE contiguous substring. When an object splits a phrasal verb, put the verb in the mention's `text` and the stranded particle in its `particle`: "The man picks a red bag up from the ground." -> mention {"text": "picks", "particle": {"text": "up", "occurrence": 1}}, objects ["a red bag"] When the verb and particle are adjacent, keep them together in `text` ("picks up") and set particle's text to null. Never return a non-contiguous string in any field. Do not use `particle` for an ordinary direction or preposition: "The man drives up the hill." -> "drives", NOT separated particle "up" "He looks at the dog." -> "looks", NOT separated particle "at" ## Subjects and objects - When the participant is an annotated entity, echo one of your top-level entity mentions -- the mention FROM THE CLAUSE WHERE THE ACTION IS FIRST MENTIONED, with exactly the occurrence index the top-level entity mentions array shows for it. In later sentences that is usually a pronoun: "A man grabs a stick. He flips." with g0: "A man"#1 | "He"#1 -> "grabs" has subject {"text": "A man", "occurrence": 1} "flips" has subject {"text": "He", "occurrence": 1} - The entity a spatial preposition relates the action to IS an object, even though the preposition itself stays out of the span: "He is sitting on a chair." -> span "is sitting", objects ["a chair"] "A boy sits next to a woman." -> span "sits", objects ["a woman"] "He looks at a dog." -> span "looks", objects ["a dog"] But broad scene regions are not objects: "walks in the park", "squats on the ground", "stands on the left" -> objects []. - Participants that are NOT listed in any group (e.g. "another man", body parts) are still returned, as exact substrings of the caption. The filming camera/viewpoint is the exception: it is out of scope under Camera/viewpoint exclusion. - A subject may be stated once and shared by several coordinated actions -- repeat it for each action, always as the mention from that clause. - `subject.text` is null only when the caption truly states no subject ("Feeding two birds with a spoon." -> subject null). ## More worked examples "A cat scratches a rooster." actions: "scratches" (subject "A cat", objects ["a rooster"]) "Cranes fly and walk, wings spread wide." actions: "fly", "walk", "spread" "A man prepares to throw a javelin, another man observes." actions: "throw" (subject "A man", objects ["a javelin"]), "observes" (subject "another man", objects []) "Two men squatting on the ground." actions: "squatting" (type posture) "A giraffe lifts another giraffe's leg with its mouth." actions: "lifts" (subject "A giraffe" occurrence 1, objects ["another giraffe's leg"]) "He looks at a dog and then picks up a ball." actions: "looks", "picks up" "A man falls down and a car hits him badly." actions: "falls", "hits" "A falling ball passes a fallen branch." actions: "falling", "passes" ("fallen" is NOT an action) Full example with groups. Input: CAPTION: A man in black on the black board grabs a stick tied to a rope and surfs. He flips a few times. He finally lands. GROUPS: g0: "A man"#1 | "He"#1 | "He"#2 g1: "the black board"#1 g2: "a stick"#1 g3: "a rope"#1 Output actions (occurrence counts run over the WHOLE caption, all fields abbreviated; every mentions array here has one element because no event is re-described): mentions ["grabs"] subject "A man"#1 objects ["a stick"#1] type simple mentions ["tied"] subject "a stick"#1 objects ["a rope"#1] type passive mentions ["surfs"] subject "A man"#1 objects [] type simple mentions ["flips"] subject "He"#1 objects [] type simple mentions ["lands"] subject "He"#2 objects [] type simple Note: "to a rope", "a few times" and "finally" are excluded from the action span. "the black board" is where the man stands; no action selects it as an object here, so g1 appears in no entry. ## Fields mentions every expression of this one action event, in caption order. Usually one element. Each mention has: text the exact substring, copied character for character from the caption occurrence 1 if this is the first occurrence of that exact substring in the caption, 2 for the second, and so on -- counted over the WHOLE caption, even for text inside other words. Repeated verbs and pronouns make this matter: in "He waves. He waves again." the second "waves" has occurrence 2. particle a separated phrasal-verb particle ({"text": null, "occurrence": 1} when there is none). See "Separated particles". type one of: simple, phrasal, prepositional, intention, light_verb, nonvisual_object, posture, passive, other subject the noun phrase performing the action, as an exact substring; text may be null when the caption states no subject objects the concrete visible entities the action involves: things it is directed at, and the entity a spatial preposition relates it to (the chair sat on). Each an exact substring; empty list when there are none. Action nouns included in a light-verb span are not objects. confidence high, medium or low. Use low when the caption is ungrammatical or the action / particle / semantic judgement is genuinely unclear. ## Hard requirements - Every `text` must appear VERBATIM in the caption. Do not normalise, lemmatise, correct spelling, expand contractions, or change capitalisation. - Every span must be contiguous; separated particles go in `particle`. - Action mentions must not overlap each other or any particle -- one stretch of the caption expresses one mention of one action. - Two mentions belong to the same entry ONLY if they describe the same single event; "again" / "once more" / a later repetition is a NEW entry. - Process every sentence; do not stop after the first. - If the caption states no action at all, return {"actions": []}. Do not invent one. - Captions are sometimes ungrammatical. Extract what is actually there and mark confidence low. Most importantly: - POS tags identify candidates; they do not determine actionhood by themselves. - Extract the actual semantic action, not every grammatical verb. - The action span is the VERB GROUP (VB* auxiliary chain + core action verb) + genuine PHRASAL PARTICLE; include the ACTION-NOUN PHRASE for light-verb or activity-noun constructions. Modals, auxiliary do and control verbs are out. - Do not include ordinary directions or prepositions. - Entity nouns/pronouns stay out; action nouns stay in light-verb constructions. - `does a flip` -> `does a flip` · `do squats` -> `do squats` - `performs jumps` -> `performs jumps` · `gives a thumbs-up` -> `gives a thumbs-up` - `plays golf` -> `plays golf` · `plays the piano` -> `plays`, objects [`the piano`] - `is walking` -> `is walking` · `was kicked` -> `was kicked` · `can swim` -> `swim` - `keeps chasing` -> `chasing` · `starts running` -> `running` - `look at it` -> `look` · `look to the left` -> `look` - `is sitting on the chair` -> `is sitting` - `pick up the ball` -> `pick up` · `turn on the light` -> `turn on` - `try to jump` -> `jump` · `prepare to throw` -> `throw` · `run to hit` -> `run`+`hit` - `falling` can be an action; result-state `fallen` is not. ## Camera/viewpoint exclusion Describe only visible people, animals, and physical objects, together with their visible actions, processes, or states. Do not describe the filming camera or camera viewpoint as a scene event, and do not put camera/viewpoint motion into the caption or actions: "the camera moves/pans/zooms/tracks/shakes" is out of scope. If that is the only apparent motion, describe the visible scene without mentioning the camera and return actions []. This excludes the filming camera/viewpoint only; a physically visible camera device may still be described when it is an ordinary scene object that is handled, moved, or dropped. ## Auxiliary do and lexical start/stop Auxiliary do/does/did is excluded from action spans. Match its function, not just its spelling: "He did fall." -> "fall"; "He does not move." -> "move". This span rule does not claim that a negated action happened in the video. Lexical or light-verb do is not auxiliary do: follow the action-noun rule, e.g. "He does a flip." -> "does a flip", type light_verb, objects []. Apply the same light-verb rule across tense, person/number, and auxiliary-chain variants: "He does sports" -> "does sports"; "They did sports" -> "did sports"; "She is doing sports" -> "is doing sports"; "He has done sports" -> "has done sports"; "They have been doing push-ups" -> "have been doing push-ups". Preserve the exact surface form. Only auxiliary do is stripped, as in "He did fall" -> "fall". Lexicalized physical-event complements stay in the action span when needed to name the event. In particular: "loses balance" -> "loses balance" and "loses its balance" -> "loses its balance", with objects []. Apply the same rule to tense and auxiliary variants such as "lost balance" and "is losing balance"; do not reduce them to only the verb. When start/stop frames another action, extract the embedded action only: "He starts running." -> "running"; "She stops walking." -> "walking"; "He starts to run." -> "run". When start/stop directly acts on a concrete entity, it is the lexical action: "He starts the engine." -> "starts", objects ["the engine"]; "She stops the car." -> "stops", objects ["the car"]. Apply this to inflected forms too: "started the machine" -> "started"; "stopped the vehicle" -> "stopped". Standalone start/stop is retained only if it expresses a clear physical event; if the text does not identify an action sufficiently, omit it or use low confidence. ## Joint response consistency Return ONE JSON object with caption, mentions, actions, and each action's temporal field in that order. The top-level mentions array contains ENTITY mentions; each action's mentions contains ACTION mentions. There is no supplied caption: all action substrings must refer to your own final caption in this response. All occurrence indices use that same caption. Do not output character offsets or invented group IDs. Local code derives groups from entity mark_ids and links action participants to those groups. For annotated action participants, reuse the exact (text, occurrence) in your entity mentions, including later pronouns. For a collective participant, create one collective entity mention linked to every relevant mark, and reuse that span. Use the visible event evidence to write the caption, then annotate every action expressed in it. Do not invent events just to populate actions. Ordinary positions, visibility and result adjectives such as "parked truck" are not action events. Do not add phase, state, or polarity. For every action, also fill its temporal field using the clean-frame sequence below. Return actions [] if the caption contains no action. # TEMPORAL LOCALIZATION IN THE SAME CALL You localize EXISTING caption actions in a VIPOSeg clip. Do not rewrite the caption, add actions, or assume the caption is visually correct. The clean images form an ordered image sequence, not an encoded video. Every saved frame is provided (nominal 2 FPS), labelled with its zero-based frame_index, timestamp, and original source filename. Additional MARK REFERENCE images identify entities; they are not additional moments. gN -> mark_id maps caption groups to these marks. Use clean frames to judge motion and reference overlays to identify the subject. Camera movement alone is not evidence of an object's action. For every action in the final actions array, return exactly one temporal object in the same order: - status: visible, not_visible, or uncertain. - segments: start_frame and end_frame, ZERO-BASED indices into the saved frame sequence, BOTH INCLUSIVE. Use the first/last frames where the specified action is visibly occurring. A single-frame event may have start_frame == end_frame. Separate intervals when the action stops and resumes. Do not merge a genuine pause merely because it lasts less than a second. Cover ongoing human postures (standing, leaning, sitting) over the observed period where they hold. - confidence: high, medium, low. This is your judgement, not calibrated accuracy. - reason: one brief English sentence describing visible evidence or uncertainty. not_visible must have segments [] and confidence low. uncertain may have no segments if location cannot be justified; a tentative interval must have low confidence. Never assign [first,last] simply to fill an action's entry. Static placement of an inanimate object (e.g. "an airplane sits across a road") is not the bodily action of sitting: return not_visible for that action, with a reason. For correctly supplied human postures, retain the existing annotation. For a group subject, localize participation by the referenced members; if a collective claim is only partly supported, say so and lower confidence. Use the whole caption to recover directional/particle context omitted from a bare action verb. Action-noun light-verb phrases are ONE action (does a flip, performs jumps, gives a thumbs-up), not separate intervals for their nouns. Only report observed evidence inside this clip. At frame 0 or the final frame, the action may already be ongoing or continue beyond the clip: do not claim its true onset/offset is known. Sparse saved images cannot establish unseen motion between images. Use the frame timestamps, especially across missing source frames. Never equate a source filename number with a saved frame_index. Put this result in the action's temporal field. Do not create a separate top-level temporal array.