ESP32-S3 AI Voice Conversation Prototype

I want to turn an on-screen AI voice conversation into a physical prototype people can actually operate. The project now has a minimal but complete push-to-talk, half-duplex interaction loop: a person holds the button and speaks, the ESP32-S3 streams audio to a cloud real-time voice session, and the device responds through its screen and speaker.

Final interaction flow

1. Press the button.
2. The device starts recording.
3. The device continuously sends microphone audio.
4. The same real-time voice session handles recognition and response generation.
5. The device continuously receives reply audio and writes it to the playback buffer.
6. The screen displays the interaction status, and the speaker plays the answer.

Current progress

As of September 22, 2026, the voice conversation goal for this phase is 100% complete. The reference device now supports button-triggered recording, real-time audio upload, cloud speech recognition and response generation, buffered reply playback, and on-screen status feedback. In one real conversation turn, the segmented-enqueue and backpressure implementation processed all 325,552 bytes of reply audio, with a queue peak of 196,607 bytes and 0 underruns. The complete answer played through, the screen showed no errors, and the audible noise was effectively gone in that test.

This is still a single Candidate acceptance test on one fixed reference device. It does not establish a stable Release, cross-device validation, or a setup that every user can reproduce directly. “Completed” means the minimal voice conversation goal defined for this phase has been achieved; installation tooling and formal release gates continue as separate open-source delivery work.

The move from staged voice processing to a real-time session—and the work of coordinating audio, networking, memory, display, and playback on the ESP32—is documented in “I Finally Got Real-Time Voice Conversation Working on an ESP32-S3”.

I connected the 4Ω/3W enclosure speaker to the MAX98357A, then connected the amplifier to the ESP32-S3 with five wires: VIN → 3V3, GND → GND, BCLK → GPIO16, LRC → GPIO17, and DIN → GPIO18. During a brief power-on check, I observed no obvious problems. A low-volume test program then produced three tones as expected. After that, the device used Wi-Fi to play five web-generated voice clips in sequence. These results show that the current output chain works, but only as a single recorded verification run in this specific setup.

The dynamic browser-to-speaker pipeline, the end-to-end evidence from five tasks, and the current limitations are documented in “Type Text in a Browser, Hear It from an ESP32 Speaker: Testing a Local TTS Playback Pipeline”.

On the platform side, reusable MP3 files and individual device playbacks are now separate objects. The same text and generation configuration can reuse an existing audio asset, while every send still creates an independent playback job and event record. This reduces repeated TTS cost and waiting time without losing each device action. The data model, cache key, and legacy-data migration are documented in “Why the Same Text Shouldn’t Generate Another MP3: Separating Audio Assets from Playback Jobs”.

The new pre-soldered INMP441 is connected as SCK → GPIO4, WS → GPIO5, SD → GPIO6, and L/R → GND. Early on, keeping I²S reads and screen refreshes in the same main loop produced only 5,120 samples during a 3.304-second operation, revealing that screen transfers were blocking capture. After moving continuous I²S reads to a dedicated FreeRTOS task, the final combined test recorded 61,184 samples in 3,827 ms. At 16 kHz, the theoretical count is about 61,232, a difference of roughly 0.08%. The screen waveform now updates locally at about 25 FPS, and direct observation confirmed that it looks noticeably smoother.

The complete investigation—from the Base64 buffer error to discovering that screen redraws were blocking capture—is documented in “Severe Undersampling in ESP32 Recordings: Why ST7789 Refreshes Starved I²S Capture”.

The wiring, capture scheduling, WAV processing, authentication checks, and web results for recording upload and transcription are documented in “ESP32-S3 + INMP441 voice input test: from press-and-hold recording to web transcription”.

The earlier recording-upload, web-transcription, and standalone-playback paths remain as verified learning stages, but the final runtime no longer chains them together as a series of separate requests. The ESP32-S3 now connects directly to a real-time voice provider and handles audio upload, recognition, response generation, and reply-audio download in one persistent session. That completes the voice conversation goal originally defined for this prototype.

Issues encountered so far

When I first prepared to connect the microphone and speaker, I discovered that unsoldered pin headers and incompatible connectors could make a hardware list that looked complete on paper unusable in the real build. I recorded that experience in “AI gave me a shopping list, but I spent a day buying parts.”.

The speaker’s white two-pin connector could not plug directly into the MAX98357A’s green screw terminal. I had bought a matching socket, but there was nowhere to use it in this assembly. I eventually removed the metal terminals from the white housing and secured them in the amplifier’s terminal block. The complete wiring process, three-tone test, and five web-generated voice clips are documented in “ESP32-S3 + MAX98357A Wiring: Play Voice Through a 4Ω/3W Speaker”.

After I connected the ST7735 display, fixed colored lines appeared along the right and bottom edges. By leaving the wiring and drawing code unchanged and switching only the panel initialization parameters, I confirmed that this screen needs INITR_GREENTAB. The full troubleshooting process is in “Colored artifact lines at the edge of an ST7735 screen: How to choose between INITR_BLACKTAB and GREENTAB”.

The GMT130-V1.0 ST7789 display I connected earlier had a backlight but no pixel image, and I once thought it was damaged. Later, I kept the original pins, ST7789 driver, and 240×240 resolution, switched to an ESP32 hardware SPI path that actually applied SPI_MODE3, and set the clock to 1 MHz. The screen completed a five-color cycle and passed a power-cycle retest. The full investigation is in “ST7789 Backlight but No Image: GMT130-V1.0 Works After Switching from SPI_MODE0 to MODE3”.

To make the GMT130 wiring easier to understand, I worked through the roles of all seven pins and changed the wiring to SCK → GPIO9, SDA → GPIO10, RES → GPIO11, and DC → GPIO12, with VCC and BLK connected to 3V3 and GND connected to GND. Learning the underlying concepts took about two days; rewiring took five minutes. After power-up, the display still cycled through the colors correctly. I explain how those seven wires work together in “ESP32 with an ST7789 screen: what do the 7 wires each do?”.

When preparing to connect the INMP441, I skipped the practice board and went straight into the first soldering job of my life. The six joints came out in inconsistent shapes, and I still need to determine whether the module will work reliably. The full process is in “The tutorial made sense, but my first solder joint still ended up as a blob.”.

These problems did not stop the project. Instead, they gradually broke the vague goal of “building a voice robot” into concrete tasks I could connect, test, and verify one at a time.