The Science Behind It All

This page starts with the code, not the papers. LSS today is a packetized signed-action prototype: it creates canonical payload bytes, signs them, fragments them, gives those fragments to visual and audio carrier experiments, reconstructs candidate fragments, and resolves the verified payload into an app action.

The research matters because it explains why media side channels are plausible. The current implementation is smaller and more explicit than the research systems: deterministic carriers, base64url fragments, FNV-1a checksums, one XOR parity fragment, Ed25519 signatures, and a simulated reader recovery pool.

How media can carry data

Digital media is already a stream of measurements. A video frame is a grid of pixels, and each pixel has color-channel values. An audio track is a sequence of samples, and those samples can be analyzed for energy at particular frequencies. If the sender changes those measurements in a controlled way, and the receiver knows where and when to look, a small side channel can ride inside ordinary media.

The trick is to make the side channel strong enough for a machine to recover, but subtle enough that the primary video or audio still feels normal. LSS uses that idea in two families:

• light carriers, where a frame or a pair of frames encodes bits through small visual differences;
• sound carriers, where short tone symbols encode bits through frequency choices.

This is the same broad reason QR codes, visible-light communication, audio modems, and screen-camera systems work: a transmitter produces a pattern, a noisy physical or media path distorts it, and a receiver estimates which symbols were probably sent.

What is implemented today

The stable implementation boundary lives in five builder-facing packages:

The web app now acts mostly as the interactive adapter: it owns routes, browser media permissions, UI, logging, and product flow coordination. The carrier algorithms it calls live in @lss/carrier.

Behind the scenes

The protocol starts with a payload object containing fields such as action, body, issuedAt, expiresAt, issuerId, mode, nonce, and payloadId. Before signing, @lss/protocol serializes the payload with stable sorted JSON keys. That makes the signature input deterministic: the same payload has the same canonical bytes.

signPayload signs those canonical bytes with Web Crypto Ed25519 and stores the signature as base64url text with signatureAlg: "ed25519". verifySignedPayload rejects any non-Ed25519 algorithm and verifies the signature over the same canonical payload bytes.

The signed envelope is then split into LsspFragment records. Each fragment includes:

The reader side is deliberately stricter than detection. A camera frame, file scan, or audio scan can only produce candidate observations. The recovery pool still has to reject bad checksums, group fragments by messageId, keep the highest-confidence observation for each fragment key, and reconstruct bytes only when enough valid fragments exist.

How light carriers work

At the package layer, the visual embedder is intentionally simple. createVisualFramePlan maps each character in fragmentData to a timed frame. Every generated frame contains four rectangular regions near the bottom of the frame. The regions receive a small luma delta based on the character code parity, so the plan says: at this time, nudge these regions brighter or darker.

That is not a complete video watermarking codec. It is a deterministic frame plan that gives the app a repeatable schedule for where visual energy should change.

@lss/carrier contains the more concrete pixel carrier. It treats a region of the video as a 24 by 24 grid, so each frame has 576 cells. frameChunkToGridBits writes a small frame format into those cells: a magic value, chunk index, total chunk count, bit length, 512 data bits, and a checksum. This is packet-like again: each visual frame carries a numbered slice of text.

For spatial reads, each grid cell stores one bit by changing the blue channel differently on the left and right halves of the cell. readCarrierCell sums blue-channel values on both halves and returns 1 when the left side is stronger than the right side. For temporal reads, carrier code compares the same cell across two frames. readTemporalCarrierCell sums the blue channel in the first frame and the second frame and decodes the bit from which frame has more blue energy.

This is blue-channel modulation, not a separate blue lamp or visible blue overlay. In the current carrier, modulateCarrierCell adjusts only the RGB blue component of pixels inside each cell, using a small default delta of 28 and clamping values to the normal 0 to 255 byte range. The scanner then compares relative blue energy: left half versus right half for a spatial read, or first frame versus second frame for a temporal read. Using one color channel keeps the prototype easy to measure and debug, and it mirrors research directions such as DeepLight that study blue-channel modulation, but it is still a prototype engineering choice that needs real media benchmarking.

This is why screen-camera research is relevant. HiLight shows that subtle display changes can carry data to a camera. DeepLight is relevant because it studies real-world display capture, flicker, blue-channel modulation, learned decoding, and screen extraction. LSS does not implement those full systems today, but @lss/carrier explores the same physical fact: small visual changes can be machine-readable.

How sound embedding works

Sound carriers use the same idea with frequency instead of pixels. A sender chooses a short tone from a known set. A receiver measures the audio samples and asks which expected frequency has the strongest energy.

At the package layer, createTonePlan maps each character to one of four ultrasonic frequencies: 16200, 16700, 17200, or 17700 Hz. renderTonePlan turns those symbols into low-amplitude sine waves at amplitude 0.02. That produces a deterministic ultrasonic plan for fragment text.

@lss/carrier has two audio carrier shapes:

The scanner uses tone-energy detection. For each symbol window, it computes energy at the expected frequencies and chooses the strongest candidate. This is close to frequency-shift keying in spirit: data is represented by switching between known frequencies over time.

Acoustic data-transmission research is relevant here because speakers, microphones, rooms, compression, and device filters all distort tones. LSS does not implement receiver collaboration or an advanced acoustic modem today. It implements a small deterministic carrier that is good enough for prototype evidence.

Are the carriers imperceptible?

Not proven yet. The current prototype tries to make carriers hard to notice, but the docs should not claim humans cannot see or hear them until LSS has perceptual testing across real devices, displays, speakers, headphones, compression paths, and viewers.

For the visual path, LSS reduces visibility by changing only one RGB channel, using small deltas, comparing relative blue energy instead of drawing visible symbols, and spreading bits across small cells and repeated frames. That makes the carrier subtler than a visible QR code, but it can still become visible if the delta is too high, the source content is flat, the display is bright, compression exaggerates artifacts, or a viewer inspects frames closely.

For the audio path, LSS uses low-gain tones in high-frequency bands. The package-level tone plan uses 16_200 to 17_700 Hz, while @lss/carrier uses marker tones at 17_400 and 18_600 Hz and text tones starting at 14_000 Hz. Those choices push energy toward the edge of many listeners’ hearing and keep amplitude low, but they do not guarantee inaudibility. Some people, devices, speakers, microphones, pets, rooms, or compression pipelines may expose artifacts.

The right claim today is narrower: LSS is experimenting with subtle visual and ultrasonic-ish carriers, and it needs benchmarking plus human perceptual studies before it can promise that ordinary viewers will not notice the carrier.

Why checksums and signatures are separate

LSS uses checksums and signatures for different jobs.

The checksum is a fast damage filter. The code uses 32-bit FNV-1a over messageId, fragmentIndex, fragmentCount, parityIndex, and decoded fragment bytes. It helps reject corrupted observations before reconstruction. It is not treated as security. RFC 9923 describes FNV as a fast non-cryptographic hash, which is exactly the role it has here.

The signature is the authenticity boundary. Ed25519 is useful because signatures are compact enough for a media-carried action envelope while still giving strong issuer authenticity. LSS follows that model in code: the resolver should only receive payloads after signature verification and app policy checks.

Checksums answer: did this fragment arrive as encoded? Signatures answer: did the issuer authorize this payload?

Why redundancy looks packet-like

Media carriers are lossy. A reader may see missing frames, duplicate observations, compression artifacts, camera blur, audio noise, or out-of-order candidates. LSS treats those observations like packet fragments.

fragmentBytes splits the signed envelope into fixed-size chunks. When there is more than one data fragment, it adds one XOR parity fragment by default. During reconstruction, the recovery pool can recover the original bytes when all data fragments are present, or when exactly one data fragment is missing and the parity fragment is present.

That is intentionally modest. It is not Reed-Solomon, fountain coding, or multi-loss forward error correction. It is one cheap parity fragment for one missing data slice.

What is future work

The current implementation should not be described as solved robust hidden-video communication. It is a working protocol boundary plus prototype carriers.

Future work includes:

• replacing the toy visual frame plan with a measured carrier codec;
• benchmarking against recompression, cropping, scaling, screen recording, brightness shifts, display refresh behavior, camera motion, and audio noise;
• hardening audio beyond the current fixed-tone prototypes, including the package-level four-frequency plan and @lss/carrier marker/text FSK scans;
• adding stronger forward error correction for multiple missing fragments;
• fusing visual, audio, camera, and file evidence into a real confidence model;
• enforcing expiry, issuer policy, key rotation, revocation, replay policy, and abuse controls around resolver usage;
• evaluating whether learned watermarking, translucency changes, blue-channel modulation, temporal frame differencing, explicit fiducials, or a hybrid carrier is the right long-term visual path.

Research context

The implementation is inspired by a few research families, but it does not copy any one paper end to end.

RivaGAN shows why surviving compression, scaling, and redistribution is hard. LSS does not implement learned video watermarking, but the future benchmarking work should be shaped by those failure modes.

HiLight is relevant because it demonstrates hidden screen-camera communication through subtle visual changes. LSS does not implement HiLight’s alpha-channel scheme, but the visual carrier direction is in the same broad family.

DeepLight is especially relevant to future visual work because it focuses on real-world displays, learned decoding, blue-channel modulation, flicker reduction, and screen extraction. LSS currently has a much simpler deterministic blue-channel carrier in @lss/carrier, but it does not implement DeepLight’s DNN decoder or object-detection screen extraction pipeline.

Acoustic data-transmission research is relevant to the audio path because it treats speakers and microphones as a noisy data channel. LSS does not implement receiver collaboration today; it renders package-level tone plans and scans deterministic marker/text tones in @lss/carrier.

For the full bibliography, see Research References.