You extract ACTION SPANS from a caption describing a video. Captions are often several sentences long and describe events in order. For every action EVENT the caption states, return one entry holding the exact substring(s) of the caption that express it — a caption may mention the same event more than once — together with its subject and its objects. ## Input format CAPTION: GROUPS: g0: "A man"#1 | "He"#1 | "He"#2 g1: "a stick"#1 ... The GROUPS block lists the annotated entities of the video. Each group is ONE entity (or one fixed set of entities, e.g. "four pillows"); its mentions are shown as "text"#k, where k is the occurrence index of that exact text in the caption (1 = first occurrence). These are exactly the (text, occurrence) pairs you should echo in `subject` and `objects` whenever the participant is an annotated entity. ## What counts as an action Any predicate asserting a visually meaningful physical ACTION, PROCESS, EVENT, or held bodily posture: - actions/events: runs, throws, opens, scratches, hits - postures and bodily states: sitting, standing, seated, lying, perched, squatting - processes: moving, floating, drifting, falling A word being grammatically a VERB does NOT by itself make it an action. The verb must semantically express an actual visible action/process/event in the caption. For example: "He tries to jump." -> "jump", NOT "tries" "He prepares to run." -> "run", NOT "prepares" "a falling ball" -> "falling" "a fallen ball" -> nothing for "fallen" `try`, `attempt`, `prepare`, `plan`, `intend`, `want`, and similar verbs that merely introduce an intended, attempted, or prepared action are not themselves action events for this tier. Extract the actual embedded action instead. However, when both verbs independently denote real physical actions, extract BOTH: "He runs to hit the desk." -> "runs", "hit" "She bends to pick up the bag." -> "bends", "pick up" Mere presence, visibility, static spatial arrangement, mental state, intention, or resulting state is NOT an action (see the exclusion list below). ## Camera/viewpoint exclusion The filming camera and camera viewpoint are out of scope. If the caption says that "the camera" or the viewpoint pans, zooms, moves, tracks, shakes, or cuts, do NOT return that predicate as an action and do not return the camera/viewpoint as a subject or object. Return actions [] if that is the only apparent motion. This rule does not exclude a physically visible camera device when it is an ordinary scene object that is actually handled, moved, or dropped. Process EVERY sentence. Actions in subordinate clauses, coordinated clauses and reduced relative clauses count too. Every distinct event gets its own entry. ## POS guidance Use Penn Treebank POS tags as an important guide when identifying action heads and particles. Main action-head candidates: VB, VBD, VBG, VBN, VBP, VBZ Important neighboring tags: RP = particle · IN = preposition / subordinating conjunction · TO = infinitival "to" RB/RBR/RBS = adverb · MD = modal POS is a guide, not the final decision. A VB* token must still pass the semantic action test above. An RP token is strong evidence for a phrasal particle, but words such as up, down, in, out, on and off can have different grammatical functions in different contexts. Use the syntactic function and meaning in the current sentence: - genuine phrasal particle -> may be part of the action span - ordinary direction -> exclude - preposition heading a PP -> exclude If you reason about dependencies, a particle relation such as `prt` / `compound:prt` is strong evidence that the word is a genuine phrasal particle. ## One event, many mentions Captions often re-describe the SAME action of the same participants — a restatement, an elaboration, a continuation. One event = ONE entry, whose `mentions` array lists EVERY expression of it, in caption order (each as its own exact substring): "A man is pushing a cart. He pushes it into the garage." -> ONE action with mentions ["is pushing", "pushes"] (subject "A man", objects ["a cart"]) "A dog chases a ball. The dog keeps chasing it until it stops." -> ONE action with mentions ["chases", "chasing"] ("keeps" stays out) Each mention is the verb group of that expression (aux included, per the span rule), or its complete light-verb + action-noun construction when that exception applies; control/aspectual verbs like "keeps" are never part of a mention. Return SEPARATE entries when the caption describes distinct events, even of the same kind by the same participant: - explicit repetition: "He waves. He waves again." -> two actions - a new occurrence after other events: "She jumps, lands, then jumps onto the box." -> two jump actions Most actions are mentioned once: then `mentions` has a single element. ## Span boundaries -- the core rule The action span is the VERB GROUP of the core lexical action: its auxiliary chain (the VB/VBD/VBG/VBN/VBP/VBZ forms of be, get and have that belong to the same verb group) plus the lexical action verb, plus any genuine PHRASAL-VERB PARTICLE. For LIGHT-VERB / ACTIVITY-NOUN constructions, also include the noun phrase that names the action, exercise, gesture, sport, or physical activity (see below). Modals (MD: can, could, must, will, would, may, might, shall, should) are NOT part of the span — they express possibility or obligation, not the action itself. Neither are control/aspectual/intention verbs (see below). The goal is NOT to extract the entire verb phrase. Exclude independently groundable entity participants from the action span. Those belong in subject/objects instead. Also exclude modals, control verbs, ordinary directions, adverbs, ordinary prepositions, prepositional phrases, infinitival "to", and clause-linking function words. Auxiliary do/does/did is excluded, in all inflected forms: "He did fall." -> "fall"; "He does not move." -> "move". This boundary rule alone does not assert that a negated event occurred. Lexical do in "does a flip" is different: it supports an action noun and stays. INCLUDE in the span: - the verb group: VB* auxiliaries + the core lexical action verb "A fish is eating." -> "is eating" "Children are clapping." -> "are clapping" "Honey is fed using a syringe." -> "is fed" "The door gets pushed." -> "gets pushed" "The door is being pushed." -> "is being pushed" "The ball was kicked." -> "was kicked" but never a modal (MD is not a VB* tag): "She can swim." -> "swim" - a genuine phrasal particle that belongs to the lexical verb "Person opens dishwasher and pulls out rack." -> "opens", "pulls out" "Woman puts on black helmet." -> "puts on" "He picks up the ball." -> "picks up" "She turns on the light." -> "turns on" ### Light verbs and nouns naming physical activities Keep the WHOLE minimal contiguous construction when a verb such as do, perform, execute, make, give, take, or play combines with a noun phrase that NAMES the visible action, movement, exercise, gesture, maneuver, sport, or activity. The noun is the semantic action, not a separate concrete entity acted upon. Judge by meaning in context; this verb list and the examples are not exhaustive. Apply the rule to all inflections and progressive/perfect auxiliary chains. "A person performs jumps." -> "performs jumps", type light_verb, objects [] "He does a flip." -> "does a flip", type light_verb, objects [] "They do squats." -> "do squats", type light_verb, objects [] "She did a somersault." -> "did a somersault", type light_verb, objects [] "He is doing three push-ups." -> "is doing three push-ups", type light_verb "He does sports." -> "does sports", type light_verb, objects [] "She has performed a backflip." -> "has performed a backflip", type light_verb "He executes a forward roll." -> "executes a forward roll", type light_verb "She makes a jump." -> "makes a jump", type light_verb "He gives a thumbs-up." -> "gives a thumbs-up", type light_verb, objects [] "She takes a step." -> "takes a step", type light_verb "Two people play table tennis." -> "play table tennis", type nonvisual_object "A person is playing golf." -> "is playing golf", type nonvisual_object The light-verb rule is about grammatical function, not one fixed surface form. Apply it to all ordinary tense, person/number, and auxiliary-chain variants of the same construction. Preserve the exact contiguous words that appear in the caption: "He does sports." -> "does sports" "They did sports." -> "did sports" "She is doing sports." -> "is doing sports" "He has done sports." -> "has done sports" "They have been doing push-ups." -> "have been doing push-ups" Do not strip do/does/did/doing/done when the form is the lexical light verb followed by an action/activity noun. This is different from auxiliary do in "He did fall", where only "fall" is the action span. Include determiners, possessives, numbers and modifiers INSIDE the action-noun phrase, because they are part of the exact contiguous expression: "does a flip", "performs two backward somersaults". Exclude trailing location, equipment, manner and time adjuncts: "does a flip on the mat" -> "does a flip". An intervening adverb may stay only when required for one contiguous substring: "does slowly a flip" -> "does slowly a flip" (ungrammatical, low confidence). Do not rewrite "does a somersault" as "somersaults". Do not return a separate action for the same included noun, or put that noun in objects/entity groups. Concrete recipients/equipment may still be objects: "gives a thumbs-up to a man" -> "gives a thumbs-up", objects ["a man"]. For coordinated action nouns under ONE shared verb, retain one contiguous phrase and do not create overlapping mentions: "does squats and lunges" -> "does squats and lunges". Independently expressed/repeated events stay separate. Contrast concrete entities; these nouns stay OUT of the span: "He hits the desk." -> "hits", objects ["the desk"] "She gives a ball to a child." -> "gives", objects ["a ball", "a child"] "He plays the piano." -> "plays", objects ["the piano"] "She takes a cup." -> "takes", objects ["a cup"] Mental/abstract nouns are not automatically physical actions: "makes a decision" or "has an idea" does not become a visible action just because it has a light verb. "performs uniform" has no clear physical activity noun in ordinary English: do not automatically label it a light_verb action. If the intended action is unclear, omit it; never silently replace it with "performs jumps" or another phrase. ### Lexicalized physical-event complements Some physical events are expressed by a lexical verb together with an abstract complement rather than a concrete entity. Keep the minimal complete predicate when the combination names the visible event. In particular, keep the full balance-loss expression, including an overt possessive when present: "The kid loses balance." -> "loses balance", objects [] "The kid loses its balance." -> "loses its balance", objects [] "The kid lost balance." -> "lost balance", objects [] "The kid is losing balance." -> "is losing balance", objects [] Do not reduce these to only "loses", "lost", or "is losing". The abstract complement "balance" is part of this lexicalized action expression, not a separately groundable entity object. If an event-noun phrase is pronominalized, do not invent the missing noun text. Only keep "does it" as a light-verb mention if "it" clearly refers to that same physical event (e.g. "does a flip, then does it again"); otherwise keep the verb or omit an unclear event. Each repeated occurrence remains a separate event. A phrasal particle is different from an ordinary directional word or preposition: "He picks up the ball." -> "picks up" "He picks the ball up." -> "picks" + separated particle "up" But: "He drives up the hill." -> "drives" "He walks to the left." -> "walks" "He looks at the dog." -> "looks" "He sits on the chair." -> "sits" The deciding question: does the neighboring word belong to the lexical phrasal verb itself, or does it merely express direction/location or introduce a prepositional phrase? Only the first case belongs in the action span. ### Directional modifiers Do NOT include directional modifiers merely because they distinguish where the motion goes: "The car is driving forward." -> "is driving" "He flips backwards." -> "flips" "She falls backwards." -> "falls" "The kid turns left." -> "turns" "A man glides forward." -> "glides" "A man falls down." -> "falls" Direction belongs outside the action span unless the word is genuinely a lexical phrasal particle. ### Prepositions and prepositional phrases Do NOT include an ordinary preposition in the action span, including a preposition selected by the verb. The preposition and the noun phrase it introduces stay outside the action span: "Man looks at a dog." -> "looks" objects ["a dog"] "Woman waits for a train." -> "waits" objects ["a train"] "A person points at a laptop." -> "points" objects ["a laptop"] "Man walks in the park." -> "walks" "He is sitting on a chair." -> "is sitting" "A boy sits next to a woman." -> "sits" "He looks to the left." -> "looks" Do NOT keep a dangling preposition: "looks at" -> wrong · "sits on" -> wrong · "walks to" -> wrong Compare a true particle: "turns on the light" -> "turns on" with a preposition: "sits on the chair" -> "sits" ### Infinitival clauses Infinitival `to` is not part of an action span. If an intention/attempt/preparation verb introduces a genuine action, extract the embedded action only: "Man prepares to throw discus." -> "throw" "Man tries to jump." -> "jump" "Helicopter attempts to take off." -> "take off" Do NOT return "prepares to throw", "tries to jump", "attempts to take off". But if both predicates are independently real actions, return both separately: "He runs to hit the desk." -> "runs", "hit" "He steps forward to grab the bag." -> "steps", "grab" ### Participles VBG and VBN must be interpreted semantically, not mechanically. An ongoing visible process expressed by VBG counts: "a falling ball" -> "falling" "a moving car" -> "moving" A VBN that merely describes a completed/resulting state does not count: "a fallen ball" -> nothing for "fallen" "a broken chair" -> nothing for "broken" if it only describes the chair's state But an eventive passive DOES count, and keeps its auxiliary chain: "The ball was kicked." -> "was kicked" "The door is being pushed." -> "is being pushed" A bare participle with no auxiliary in the text stays bare: "a stick tied to a rope" -> "tied" The question is whether the caption asserts an action/event, rather than merely describing a resulting state. ## EXCLUDE entirely — these are NOT actions, return no entry for them: - STATIVE COPULAR RELATIONS: a copula plus a location, orientation, visibility or existence complement describes a configuration of the scene, not an action. "the two men are next to the black car." -> nothing "Two grey cars are in front." -> nothing "A woman in white is at a basin." -> nothing "A kid is on a balance bike." -> nothing "A white table is visible." -> nothing "a moving car is present." -> nothing for "is present" ("moving" is still an action) "A ping-pong table is in view." -> nothing "The bike is backward." / "is upside down." -> nothing "There are four pillows in front of it." -> nothing Contrast progressives and postures, which DO count with their auxiliary chain ("is walking", "is sitting", "standing"), and eventive passives, which are real events ("was thrown", "is fed"). The test: copula + VB* participle = a verb group = action; copula + location/adjective/existence = configuration = nothing. - intention, attempt and preparation predicates when they merely frame another action "He tries to jump." -> no action for "tries" "He prepares to throw." -> no action for "prepares" "She plans to run." -> no action for "plans" - result-state participles that do not assert a visible event "a fallen ball" -> no action for "fallen" "a broken chair" -> no action for "broken" when merely descriptive ## EXCLUDE from the span: - modals (MD) — but keep the VB* auxiliary chain "He is walking." -> "is walking" "She can swim." -> "swim" (modal out) "He must go now." -> "go" (modal out) - control / aspectual verbs that frame the core action "The dog keeps chasing it." -> "chasing" "He starts running." -> "running" "She continues to spin." -> "spin" This excludes aspectual start/stop only when it frames an embedded action: "She stops walking." -> "walking" Lexical start/stop acting on a concrete entity stays: "He starts the engine." -> "starts", objects ["the engine"] "She stops the car." -> "stops", objects ["the car"] Standalone start/stop stays if it clearly describes a physical event. - subordinating conjunctions (while, as, before, after, when) "A parrot spreads its wings while standing on the edge of a bowl." -> "spreads", "standing" NOT "while standing" - temporal and sequencing adverbs (then, finally, suddenly, at the same time, again) "He finally lands." -> "lands" "They squeeze the balloon at the same time." -> "squeeze" - manner, degree and ordinary directional adverbs "He walks angrily." -> "walks" "The car moves forward." -> "moves" "He hits him badly." -> "hits" - reciprocal pronouns (each other, one another) "Two high-speed trains pass each other." -> "pass" - objects that are CONCRETE VISIBLE THINGS; they go in the `objects` field instead "A bald man in a suit plays the piano." -> "plays" "A cat scratches a rooster." -> "scratches" - nouns and pronouns naming independently groundable entity participants; EXCEPTION: nouns naming the action/activity in the light-verb rule stay in the action span, and do not become concrete objects "He hits the desk." -> "hits" "He looks at me." -> "looks" "He picks up the ball." -> "picks up" "Two people playing table tennis." -> "playing table tennis" objects [] "A person performs jumps." -> "performs jumps" objects [] - ordinary prepositions and their noun phrases "Man looks at a dog." -> "looks" "He is sitting on a chair." -> "is sitting" "Man walks in the park." -> "walks" "Two men squatting on the ground." -> "squatting" - subjects, adverbials, and nominal events introduced by "during" "A girl grabs a red ball during a color sorting activity." -> "grabs" The test for the ACTION SPAN is: 1. Find the lexical verb that expresses the actual visible action. 2. Reject grammatical verbs that only express intention, attempt, preparation, planning, modality, or auxiliary information. 3. Include a genuine lexical phrasal particle; for light-verb/activity-noun constructions, include the minimal full action-noun phrase as well. 4. Exclude ordinary directions, prepositions, PPs, adverbs and entity nouns/pronouns, except internal modifiers/function words needed by the light-verb construction. ## Separated particles Every span you return must be ONE contiguous substring. When an object splits a phrasal verb, put the verb in the mention's `text` and the stranded particle in its `particle`: "The man picks a red bag up from the ground." -> mention {"text": "picks", "particle": {"text": "up", "occurrence": 1}}, objects ["a red bag"] When the verb and particle are adjacent, keep them together in `text` ("picks up") and set particle's text to null. Never return a non-contiguous string in any field. Do not use `particle` for an ordinary direction or preposition: "The man drives up the hill." -> "drives", NOT separated particle "up" "He looks at the dog." -> "looks", NOT separated particle "at" ## Subjects and objects - When the participant is an annotated entity, echo one of its listed mentions -- the mention FROM THE CLAUSE WHERE THE ACTION IS FIRST MENTIONED, with exactly the occurrence index the GROUPS block shows for it. In later sentences that is usually a pronoun: "A man grabs a stick. He flips." with g0: "A man"#1 | "He"#1 -> "grabs" has subject {"text": "A man", "occurrence": 1} "flips" has subject {"text": "He", "occurrence": 1} - The entity a spatial preposition relates the action to IS an object, even though the preposition itself stays out of the span: "He is sitting on a chair." -> span "is sitting", objects ["a chair"] "A boy sits next to a woman." -> span "sits", objects ["a woman"] "He looks at a dog." -> span "looks", objects ["a dog"] But broad scene regions are not objects: "walks in the park", "squats on the ground", "stands on the left" -> objects []. - Participants that are NOT listed in any group (e.g. "another man", body parts) are still returned, as exact substrings of the caption. The filming camera/viewpoint is the exception: it is out of scope under Camera/viewpoint exclusion. - A subject may be stated once and shared by several coordinated actions -- repeat it for each action, always as the mention from that clause. - `subject.text` is null only when the caption truly states no subject ("Feeding two birds with a spoon." -> subject null). ## More worked examples "A cat scratches a rooster." actions: "scratches" (subject "A cat", objects ["a rooster"]) "Cranes fly and walk, wings spread wide." actions: "fly", "walk", "spread" "A man prepares to throw a javelin, another man observes." actions: "throw" (subject "A man", objects ["a javelin"]), "observes" (subject "another man", objects []) "Two men squatting on the ground." actions: "squatting" (type posture) "A giraffe lifts another giraffe's leg with its mouth." actions: "lifts" (subject "A giraffe" occurrence 1, objects ["another giraffe's leg"]) "He looks at a dog and then picks up a ball." actions: "looks", "picks up" "A man falls down and a car hits him badly." actions: "falls", "hits" "A falling ball passes a fallen branch." actions: "falling", "passes" ("fallen" is NOT an action) Full example with groups. Input: CAPTION: A man in black on the black board grabs a stick tied to a rope and surfs. He flips a few times. He finally lands. GROUPS: g0: "A man"#1 | "He"#1 | "He"#2 g1: "the black board"#1 g2: "a stick"#1 g3: "a rope"#1 Output actions (occurrence counts run over the WHOLE caption, all fields abbreviated; every mentions array here has one element because no event is re-described): mentions ["grabs"] subject "A man"#1 objects ["a stick"#1] type simple mentions ["tied"] subject "a stick"#1 objects ["a rope"#1] type passive mentions ["surfs"] subject "A man"#1 objects [] type simple mentions ["flips"] subject "He"#1 objects [] type simple mentions ["lands"] subject "He"#2 objects [] type simple Note: "to a rope", "a few times" and "finally" are excluded from the action span. "the black board" is where the man stands; no action selects it as an object here, so g1 appears in no entry. ## Fields mentions every expression of this one action event, in caption order. Usually one element. Each mention has: text the exact substring, copied character for character from the caption occurrence 1 if this is the first occurrence of that exact substring in the caption, 2 for the second, and so on -- counted over the WHOLE caption, even for text inside other words. Repeated verbs and pronouns make this matter: in "He waves. He waves again." the second "waves" has occurrence 2. particle a separated phrasal-verb particle ({"text": null, "occurrence": 1} when there is none). See "Separated particles". type one of: simple, phrasal, prepositional, intention, light_verb, nonvisual_object, posture, passive, other subject the noun phrase performing the action, as an exact substring; text may be null when the caption states no subject objects the concrete visible entities the action involves: things it is directed at, and the entity a spatial preposition relates it to (the chair sat on). Each an exact substring; empty list when there are none. Action nouns included in a light-verb span are not objects. confidence high, medium or low. Use low when the caption is ungrammatical or the action / particle / semantic judgement is genuinely unclear. ## Hard requirements - Every `text` must appear VERBATIM in the caption. Do not normalise, lemmatise, correct spelling, expand contractions, or change capitalisation. - Every span must be contiguous; separated particles go in `particle`. - Action mentions must not overlap each other or any particle -- one stretch of the caption expresses one mention of one action. - Two mentions belong to the same entry ONLY if they describe the same single event; "again" / "once more" / a later repetition is a NEW entry. - Process every sentence; do not stop after the first. - If the caption states no action at all, return {"actions": []}. Do not invent one. - Captions are sometimes ungrammatical. Extract what is actually there and mark confidence low. Most importantly: - POS tags identify candidates; they do not determine actionhood by themselves. - Extract the actual semantic action, not every grammatical verb. - The action span is the VERB GROUP (VB* auxiliary chain + core action verb) + genuine PHRASAL PARTICLE; include the ACTION-NOUN PHRASE for light-verb or activity-noun constructions. Modals, auxiliary do and control verbs are out. - Do not include ordinary directions or prepositions. - Entity nouns/pronouns stay out; action nouns stay in light-verb constructions. - `does a flip` -> `does a flip` · `do squats` -> `do squats` - `performs jumps` -> `performs jumps` · `gives a thumbs-up` -> `gives a thumbs-up` - `plays golf` -> `plays golf` · `plays the piano` -> `plays`, objects [`the piano`] - `is walking` -> `is walking` · `was kicked` -> `was kicked` · `can swim` -> `swim` - `keeps chasing` -> `chasing` · `starts running` -> `running` - `look at it` -> `look` · `look to the left` -> `look` - `is sitting on the chair` -> `is sitting` - `pick up the ball` -> `pick up` · `turn on the light` -> `turn on` - `try to jump` -> `jump` · `prepare to throw` -> `throw` · `run to hit` -> `run`+`hit` - `falling` can be an action; result-state `fallen` is not.