feat: add RunPod AI integration with image generation and enhanced LLM support
Add comprehensive RunPod AI API integration including: - New runpodApi.ts client for RunPod endpoint communication - Image generation tool and shape utilities for AI-generated images - Enhanced LLM utilities with RunPod support for text generation - Updated Whisper transcription with improved error handling - UI components for image generation tool - Setup and testing documentation This commit preserves work-in-progress RunPod integration before switching branches. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
5878579980
commit
080e5a3b87
|
|
@ -0,0 +1,255 @@
|
||||||
|
# RunPod WhisperX Integration Setup
|
||||||
|
|
||||||
|
This guide explains how to set up and use the RunPod WhisperX endpoint for transcription in the canvas website.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The transcription system can now use a hosted WhisperX endpoint on RunPod instead of running the Whisper model locally in the browser. This provides:
|
||||||
|
- Better accuracy with WhisperX's advanced features
|
||||||
|
- Faster processing (no model download needed)
|
||||||
|
- Reduced client-side resource usage
|
||||||
|
- Support for longer audio files
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
1. A RunPod account with an active WhisperX endpoint
|
||||||
|
2. Your RunPod API key
|
||||||
|
3. Your RunPod endpoint ID
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
Add the following environment variables to your `.env.local` file (or your deployment environment):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# RunPod Configuration
|
||||||
|
VITE_RUNPOD_API_KEY=your_runpod_api_key_here
|
||||||
|
VITE_RUNPOD_ENDPOINT_ID=your_endpoint_id_here
|
||||||
|
```
|
||||||
|
|
||||||
|
Or if using Next.js:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
NEXT_PUBLIC_RUNPOD_API_KEY=your_runpod_api_key_here
|
||||||
|
NEXT_PUBLIC_RUNPOD_ENDPOINT_ID=your_endpoint_id_here
|
||||||
|
```
|
||||||
|
|
||||||
|
### Getting Your RunPod Credentials
|
||||||
|
|
||||||
|
1. **API Key**:
|
||||||
|
- Go to [RunPod Settings](https://www.runpod.io/console/user/settings)
|
||||||
|
- Navigate to API Keys section
|
||||||
|
- Create a new API key or copy an existing one
|
||||||
|
|
||||||
|
2. **Endpoint ID**:
|
||||||
|
- Go to [RunPod Serverless Endpoints](https://www.runpod.io/console/serverless)
|
||||||
|
- Find your WhisperX endpoint
|
||||||
|
- Copy the endpoint ID from the URL or endpoint details
|
||||||
|
- Example: If your endpoint URL is `https://api.runpod.ai/v2/lrtisuv8ixbtub/run`, then `lrtisuv8ixbtub` is your endpoint ID
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Automatic Detection
|
||||||
|
|
||||||
|
The transcription hook automatically detects if RunPod is configured and uses it instead of the local Whisper model. No code changes are needed!
|
||||||
|
|
||||||
|
### Manual Override
|
||||||
|
|
||||||
|
If you want to explicitly control which transcription method to use:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { useWhisperTranscription } from '@/hooks/useWhisperTranscriptionSimple'
|
||||||
|
|
||||||
|
const {
|
||||||
|
isRecording,
|
||||||
|
transcript,
|
||||||
|
startRecording,
|
||||||
|
stopRecording
|
||||||
|
} = useWhisperTranscription({
|
||||||
|
useRunPod: true, // Force RunPod usage
|
||||||
|
language: 'en',
|
||||||
|
onTranscriptUpdate: (text) => {
|
||||||
|
console.log('New transcript:', text)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
Or to force local model:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
useWhisperTranscription({
|
||||||
|
useRunPod: false, // Force local Whisper model
|
||||||
|
// ... other options
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Format
|
||||||
|
|
||||||
|
The integration sends audio data to your RunPod endpoint in the following format:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"input": {
|
||||||
|
"audio": "base64_encoded_audio_data",
|
||||||
|
"audio_format": "audio/wav",
|
||||||
|
"language": "en",
|
||||||
|
"task": "transcribe"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Expected Response Format
|
||||||
|
|
||||||
|
The endpoint should return one of these formats:
|
||||||
|
|
||||||
|
**Direct Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"output": {
|
||||||
|
"text": "Transcribed text here"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Or with segments:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"output": {
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"start": 0.0,
|
||||||
|
"end": 2.5,
|
||||||
|
"text": "Transcribed text here"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Async Job Pattern:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "job-id-123",
|
||||||
|
"status": "IN_QUEUE"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The integration automatically handles async jobs by polling the status endpoint until completion.
|
||||||
|
|
||||||
|
## Customizing the API Request
|
||||||
|
|
||||||
|
If your WhisperX endpoint expects a different request format, you can modify `src/lib/runpodApi.ts`:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// In transcribeWithRunPod function
|
||||||
|
const requestBody = {
|
||||||
|
input: {
|
||||||
|
// Adjust these fields based on your endpoint
|
||||||
|
audio: audioBase64,
|
||||||
|
// Add or modify fields as needed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "RunPod API key or endpoint ID not configured"
|
||||||
|
|
||||||
|
- Ensure environment variables are set correctly
|
||||||
|
- Restart your development server after adding environment variables
|
||||||
|
- Check that variable names match exactly (case-sensitive)
|
||||||
|
|
||||||
|
### "RunPod API error: 401"
|
||||||
|
|
||||||
|
- Verify your API key is correct
|
||||||
|
- Check that your API key has not expired
|
||||||
|
- Ensure you're using the correct API key format
|
||||||
|
|
||||||
|
### "RunPod API error: 404"
|
||||||
|
|
||||||
|
- Verify your endpoint ID is correct
|
||||||
|
- Check that your endpoint is active in the RunPod console
|
||||||
|
- Ensure the endpoint URL format matches: `https://api.runpod.ai/v2/{ENDPOINT_ID}/run`
|
||||||
|
|
||||||
|
### "No transcription text found in RunPod response"
|
||||||
|
|
||||||
|
- Check your endpoint's response format matches the expected format
|
||||||
|
- Verify your WhisperX endpoint is configured correctly
|
||||||
|
- Check the browser console for detailed error messages
|
||||||
|
|
||||||
|
### "Failed to return job results" (400 Bad Request)
|
||||||
|
|
||||||
|
This error occurs on the **server side** when your WhisperX endpoint tries to return results. This typically means:
|
||||||
|
|
||||||
|
1. **Response format mismatch**: Your endpoint's response doesn't match RunPod's expected format
|
||||||
|
- Ensure your endpoint returns: `{"output": {"text": "..."}}` or `{"output": {"segments": [...]}}`
|
||||||
|
- The response must be valid JSON
|
||||||
|
- Check your endpoint handler code to ensure it's returning the correct structure
|
||||||
|
|
||||||
|
2. **Response size limits**: The response might be too large
|
||||||
|
- Try with shorter audio files first
|
||||||
|
- Check RunPod's response size limits
|
||||||
|
|
||||||
|
3. **Timeout issues**: The endpoint might be taking too long to process
|
||||||
|
- Check your endpoint logs for processing time
|
||||||
|
- Consider optimizing your WhisperX model configuration
|
||||||
|
|
||||||
|
4. **Check endpoint handler**: Review your WhisperX endpoint's `handler.py` or equivalent:
|
||||||
|
```python
|
||||||
|
# Example correct format
|
||||||
|
def handler(event):
|
||||||
|
# ... process audio ...
|
||||||
|
return {
|
||||||
|
"output": {
|
||||||
|
"text": transcription_text
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Transcription not working
|
||||||
|
|
||||||
|
- Check browser console for errors
|
||||||
|
- Verify your endpoint is active and responding
|
||||||
|
- Test your endpoint directly using curl or Postman
|
||||||
|
- Ensure audio format is supported (WAV format is recommended)
|
||||||
|
- Check RunPod endpoint logs for server-side errors
|
||||||
|
|
||||||
|
## Testing Your Endpoint
|
||||||
|
|
||||||
|
You can test your RunPod endpoint directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/run \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_API_KEY" \
|
||||||
|
-d '{
|
||||||
|
"input": {
|
||||||
|
"audio": "base64_audio_data_here",
|
||||||
|
"audio_format": "audio/wav",
|
||||||
|
"language": "en"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fallback Behavior
|
||||||
|
|
||||||
|
If RunPod is not configured or fails, the system will:
|
||||||
|
1. Try to use RunPod if configured
|
||||||
|
2. Fall back to local Whisper model if RunPod fails or is not configured
|
||||||
|
3. Show error messages if both methods fail
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- **RunPod**: Better for longer audio files and higher accuracy, but requires network connection
|
||||||
|
- **Local Model**: Works offline, but requires model download and uses more client resources
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
For issues specific to:
|
||||||
|
- **RunPod API**: Check [RunPod Documentation](https://docs.runpod.io)
|
||||||
|
- **WhisperX**: Check your WhisperX endpoint configuration
|
||||||
|
- **Integration**: Check browser console for detailed error messages
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1,139 @@
|
||||||
|
# Testing RunPod AI Integration
|
||||||
|
|
||||||
|
This guide explains how to test the RunPod AI API integration in development.
|
||||||
|
|
||||||
|
## Quick Setup
|
||||||
|
|
||||||
|
1. **Add RunPod environment variables to `.env.local`:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add these lines to your .env.local file
|
||||||
|
VITE_RUNPOD_API_KEY=your_runpod_api_key_here
|
||||||
|
VITE_RUNPOD_ENDPOINT_ID=your_endpoint_id_here
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important:** Replace `your_runpod_api_key_here` and `your_endpoint_id_here` with your actual RunPod credentials.
|
||||||
|
|
||||||
|
2. **Get your RunPod credentials:**
|
||||||
|
- **API Key**: Go to [RunPod Settings](https://www.runpod.io/console/user/settings) → API Keys section
|
||||||
|
- **Endpoint ID**: Go to [RunPod Serverless Endpoints](https://www.runpod.io/console/serverless) → Find your endpoint → Copy the ID from the URL
|
||||||
|
- Example: If URL is `https://api.runpod.ai/v2/jqd16o7stu29vq/run`, then `jqd16o7stu29vq` is your endpoint ID
|
||||||
|
|
||||||
|
3. **Restart the dev server:**
|
||||||
|
```bash
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing the Integration
|
||||||
|
|
||||||
|
### Method 1: Using Prompt Shapes
|
||||||
|
1. Open the canvas website in your browser
|
||||||
|
2. Select the **Prompt** tool from the toolbar (or press the keyboard shortcut)
|
||||||
|
3. Click on the canvas to create a prompt shape
|
||||||
|
4. Type a prompt like "Write a hello world program in Python"
|
||||||
|
5. Press Enter or click the send button
|
||||||
|
6. The AI response should appear in the prompt shape
|
||||||
|
|
||||||
|
### Method 2: Using Arrow LLM Action
|
||||||
|
1. Create an arrow shape pointing from one shape to another
|
||||||
|
2. Add text to the arrow (this becomes the prompt)
|
||||||
|
3. Select the arrow
|
||||||
|
4. Press **Alt+G** (or use the action menu)
|
||||||
|
5. The AI will process the prompt and fill the target shape with the response
|
||||||
|
|
||||||
|
### Method 3: Using Command Palette
|
||||||
|
1. Press **Cmd+J** (Mac) or **Ctrl+J** (Windows/Linux) to open the LLM view
|
||||||
|
2. Type your prompt
|
||||||
|
3. Press Enter
|
||||||
|
4. The response should appear
|
||||||
|
|
||||||
|
## Verifying RunPod is Being Used
|
||||||
|
|
||||||
|
1. **Open browser console** (F12 or Cmd+Option+I)
|
||||||
|
2. Look for these log messages:
|
||||||
|
- `🔑 Found RunPod configuration from environment variables - using as primary AI provider`
|
||||||
|
- `🔍 Found X available AI providers: runpod (default)`
|
||||||
|
- `🔄 Attempting to use runpod API (default)...`
|
||||||
|
|
||||||
|
3. **Check Network tab:**
|
||||||
|
- Look for requests to `https://api.runpod.ai/v2/{endpointId}/run`
|
||||||
|
- The request should have `Authorization: Bearer {your_api_key}` header
|
||||||
|
|
||||||
|
## Expected Behavior
|
||||||
|
|
||||||
|
- **With RunPod configured**: RunPod will be used FIRST (priority over user API keys)
|
||||||
|
- **Without RunPod**: System will fall back to user-configured API keys (OpenAI, Anthropic, etc.)
|
||||||
|
- **If both fail**: You'll see an error message
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "No valid API key found for any provider"
|
||||||
|
- Check that `.env.local` has the correct variable names (`VITE_RUNPOD_API_KEY` and `VITE_RUNPOD_ENDPOINT_ID`)
|
||||||
|
- Restart the dev server after adding environment variables
|
||||||
|
- Check browser console for detailed error messages
|
||||||
|
|
||||||
|
### "RunPod API error: 401"
|
||||||
|
- Verify your API key is correct
|
||||||
|
- Check that your API key hasn't expired
|
||||||
|
- Ensure you're using the correct API key format
|
||||||
|
|
||||||
|
### "RunPod API error: 404"
|
||||||
|
- Verify your endpoint ID is correct
|
||||||
|
- Check that your endpoint is active in RunPod console
|
||||||
|
- Ensure the endpoint URL format matches: `https://api.runpod.ai/v2/{ENDPOINT_ID}/run`
|
||||||
|
|
||||||
|
### RunPod not being used
|
||||||
|
- Check browser console for `🔑 Found RunPod configuration` message
|
||||||
|
- Verify environment variables are loaded (check `import.meta.env.VITE_RUNPOD_API_KEY` in console)
|
||||||
|
- Make sure you restarted the dev server after adding environment variables
|
||||||
|
|
||||||
|
## Testing Different Scenarios
|
||||||
|
|
||||||
|
### Test 1: RunPod Only (No User Keys)
|
||||||
|
1. Remove or clear any user API keys from localStorage
|
||||||
|
2. Set RunPod environment variables
|
||||||
|
3. Run an AI command
|
||||||
|
4. Should use RunPod automatically
|
||||||
|
|
||||||
|
### Test 2: RunPod Priority (With User Keys)
|
||||||
|
1. Set RunPod environment variables
|
||||||
|
2. Also configure user API keys in settings
|
||||||
|
3. Run an AI command
|
||||||
|
4. Should use RunPod FIRST, then fall back to user keys if RunPod fails
|
||||||
|
|
||||||
|
### Test 3: Fallback Behavior
|
||||||
|
1. Set RunPod environment variables with invalid credentials
|
||||||
|
2. Configure valid user API keys
|
||||||
|
3. Run an AI command
|
||||||
|
4. Should try RunPod first, fail, then use user keys
|
||||||
|
|
||||||
|
## API Request Format
|
||||||
|
|
||||||
|
The integration sends requests in this format:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"input": {
|
||||||
|
"prompt": "Your prompt text here"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The system prompt and user prompt are combined into a single prompt string.
|
||||||
|
|
||||||
|
## Response Handling
|
||||||
|
|
||||||
|
The integration handles multiple response formats:
|
||||||
|
- Direct text response: `{ "output": "text" }`
|
||||||
|
- Object with text: `{ "output": { "text": "..." } }`
|
||||||
|
- Object with response: `{ "output": { "response": "..." } }`
|
||||||
|
- Async jobs: Polls until completion
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
Once testing is successful:
|
||||||
|
1. Verify RunPod responses are working correctly
|
||||||
|
2. Test with different prompt types
|
||||||
|
3. Monitor RunPod usage and costs
|
||||||
|
4. Consider adding rate limiting if needed
|
||||||
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||||
import { pipeline, env } from '@xenova/transformers'
|
import { pipeline, env } from '@xenova/transformers'
|
||||||
|
import { transcribeWithRunPod } from '../lib/runpodApi'
|
||||||
|
import { isRunPodConfigured } from '../lib/clientConfig'
|
||||||
|
|
||||||
// Configure the transformers library
|
// Configure the transformers library
|
||||||
env.allowRemoteModels = true
|
env.allowRemoteModels = true
|
||||||
|
|
@ -48,6 +50,44 @@ function detectAudioFormat(blob: Blob): Promise<string> {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Convert Float32Array audio data to WAV blob
|
||||||
|
async function createWavBlob(audioData: Float32Array, sampleRate: number): Promise<Blob> {
|
||||||
|
const length = audioData.length
|
||||||
|
const buffer = new ArrayBuffer(44 + length * 2)
|
||||||
|
const view = new DataView(buffer)
|
||||||
|
|
||||||
|
// WAV header
|
||||||
|
const writeString = (offset: number, string: string) => {
|
||||||
|
for (let i = 0; i < string.length; i++) {
|
||||||
|
view.setUint8(offset + i, string.charCodeAt(i))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
writeString(0, 'RIFF')
|
||||||
|
view.setUint32(4, 36 + length * 2, true)
|
||||||
|
writeString(8, 'WAVE')
|
||||||
|
writeString(12, 'fmt ')
|
||||||
|
view.setUint32(16, 16, true)
|
||||||
|
view.setUint16(20, 1, true)
|
||||||
|
view.setUint16(22, 1, true)
|
||||||
|
view.setUint32(24, sampleRate, true)
|
||||||
|
view.setUint32(28, sampleRate * 2, true)
|
||||||
|
view.setUint16(32, 2, true)
|
||||||
|
view.setUint16(34, 16, true)
|
||||||
|
writeString(36, 'data')
|
||||||
|
view.setUint32(40, length * 2, true)
|
||||||
|
|
||||||
|
// Convert float samples to 16-bit PCM
|
||||||
|
let offset = 44
|
||||||
|
for (let i = 0; i < length; i++) {
|
||||||
|
const sample = Math.max(-1, Math.min(1, audioData[i]))
|
||||||
|
view.setInt16(offset, sample < 0 ? sample * 0x8000 : sample * 0x7FFF, true)
|
||||||
|
offset += 2
|
||||||
|
}
|
||||||
|
|
||||||
|
return new Blob([buffer], { type: 'audio/wav' })
|
||||||
|
}
|
||||||
|
|
||||||
// Simple resampling function for audio data
|
// Simple resampling function for audio data
|
||||||
function resampleAudio(audioData: Float32Array, fromSampleRate: number, toSampleRate: number): Float32Array {
|
function resampleAudio(audioData: Float32Array, fromSampleRate: number, toSampleRate: number): Float32Array {
|
||||||
if (fromSampleRate === toSampleRate) {
|
if (fromSampleRate === toSampleRate) {
|
||||||
|
|
@ -103,6 +143,7 @@ interface UseWhisperTranscriptionOptions {
|
||||||
enableAdvancedErrorHandling?: boolean
|
enableAdvancedErrorHandling?: boolean
|
||||||
modelOptions?: ModelOption[]
|
modelOptions?: ModelOption[]
|
||||||
autoInitialize?: boolean // If false, model will only load when startRecording is called
|
autoInitialize?: boolean // If false, model will only load when startRecording is called
|
||||||
|
useRunPod?: boolean // If true, use RunPod WhisperX endpoint instead of local model (defaults to checking if RunPod is configured)
|
||||||
}
|
}
|
||||||
|
|
||||||
export const useWhisperTranscription = ({
|
export const useWhisperTranscription = ({
|
||||||
|
|
@ -112,8 +153,11 @@ export const useWhisperTranscription = ({
|
||||||
enableStreaming = false,
|
enableStreaming = false,
|
||||||
enableAdvancedErrorHandling = false,
|
enableAdvancedErrorHandling = false,
|
||||||
modelOptions,
|
modelOptions,
|
||||||
autoInitialize = true // Default to true for backward compatibility
|
autoInitialize = true, // Default to true for backward compatibility
|
||||||
|
useRunPod = undefined // If undefined, auto-detect based on configuration
|
||||||
}: UseWhisperTranscriptionOptions = {}) => {
|
}: UseWhisperTranscriptionOptions = {}) => {
|
||||||
|
// Auto-detect RunPod usage if not explicitly set
|
||||||
|
const shouldUseRunPod = useRunPod !== undefined ? useRunPod : isRunPodConfigured()
|
||||||
const [isRecording, setIsRecording] = useState(false)
|
const [isRecording, setIsRecording] = useState(false)
|
||||||
const [isTranscribing, setIsTranscribing] = useState(false)
|
const [isTranscribing, setIsTranscribing] = useState(false)
|
||||||
const [isSpeaking, setIsSpeaking] = useState(false)
|
const [isSpeaking, setIsSpeaking] = useState(false)
|
||||||
|
|
@ -161,6 +205,13 @@ export const useWhisperTranscription = ({
|
||||||
|
|
||||||
// Initialize transcriber with optional advanced error handling
|
// Initialize transcriber with optional advanced error handling
|
||||||
const initializeTranscriber = useCallback(async () => {
|
const initializeTranscriber = useCallback(async () => {
|
||||||
|
// Skip model loading if using RunPod
|
||||||
|
if (shouldUseRunPod) {
|
||||||
|
console.log('🚀 Using RunPod WhisperX endpoint - skipping local model loading')
|
||||||
|
setModelLoaded(true) // Mark as "loaded" since we don't need a local model
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
if (transcriberRef.current) return transcriberRef.current
|
if (transcriberRef.current) return transcriberRef.current
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
@ -432,19 +483,33 @@ export const useWhisperTranscription = ({
|
||||||
|
|
||||||
console.log(`🎵 Real-time audio: ${processedAudioData.length} samples (${(processedAudioData.length / 16000).toFixed(2)}s)`)
|
console.log(`🎵 Real-time audio: ${processedAudioData.length} samples (${(processedAudioData.length / 16000).toFixed(2)}s)`)
|
||||||
|
|
||||||
// Transcribe with parameters optimized for real-time processing
|
let transcriptionText = ''
|
||||||
const result = await transcriberRef.current(processedAudioData, {
|
|
||||||
language: language,
|
|
||||||
task: 'transcribe',
|
|
||||||
return_timestamps: false,
|
|
||||||
chunk_length_s: 5, // Longer chunks for better context
|
|
||||||
stride_length_s: 2, // Larger stride for better coverage
|
|
||||||
no_speech_threshold: 0.3, // Higher threshold to reduce noise
|
|
||||||
logprob_threshold: -0.8, // More sensitive detection
|
|
||||||
compression_ratio_threshold: 2.0 // More permissive for real-time
|
|
||||||
})
|
|
||||||
|
|
||||||
const transcriptionText = result?.text || ''
|
// Use RunPod if configured, otherwise use local model
|
||||||
|
if (shouldUseRunPod) {
|
||||||
|
console.log('🚀 Using RunPod WhisperX API for real-time transcription...')
|
||||||
|
// Convert processed audio data back to blob for RunPod
|
||||||
|
const wavBlob = await createWavBlob(processedAudioData, 16000)
|
||||||
|
transcriptionText = await transcribeWithRunPod(wavBlob, language)
|
||||||
|
} else {
|
||||||
|
// Use local Whisper model
|
||||||
|
if (!transcriberRef.current) {
|
||||||
|
console.log('⚠️ Transcriber not available for real-time processing')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
const result = await transcriberRef.current(processedAudioData, {
|
||||||
|
language: language,
|
||||||
|
task: 'transcribe',
|
||||||
|
return_timestamps: false,
|
||||||
|
chunk_length_s: 5, // Longer chunks for better context
|
||||||
|
stride_length_s: 2, // Larger stride for better coverage
|
||||||
|
no_speech_threshold: 0.3, // Higher threshold to reduce noise
|
||||||
|
logprob_threshold: -0.8, // More sensitive detection
|
||||||
|
compression_ratio_threshold: 2.0 // More permissive for real-time
|
||||||
|
})
|
||||||
|
|
||||||
|
transcriptionText = result?.text || ''
|
||||||
|
}
|
||||||
if (transcriptionText.trim()) {
|
if (transcriptionText.trim()) {
|
||||||
lastTranscriptionTimeRef.current = Date.now()
|
lastTranscriptionTimeRef.current = Date.now()
|
||||||
console.log(`✅ Real-time transcript: "${transcriptionText.trim()}"`)
|
console.log(`✅ Real-time transcript: "${transcriptionText.trim()}"`)
|
||||||
|
|
@ -453,53 +518,63 @@ export const useWhisperTranscription = ({
|
||||||
} else {
|
} else {
|
||||||
console.log('⚠️ No real-time transcription text produced, trying fallback parameters...')
|
console.log('⚠️ No real-time transcription text produced, trying fallback parameters...')
|
||||||
|
|
||||||
// Try with more permissive parameters for real-time processing
|
// Try with more permissive parameters for real-time processing (only for local model)
|
||||||
try {
|
if (!shouldUseRunPod && transcriberRef.current) {
|
||||||
const fallbackResult = await transcriberRef.current(processedAudioData, {
|
try {
|
||||||
task: 'transcribe',
|
const fallbackResult = await transcriberRef.current(processedAudioData, {
|
||||||
return_timestamps: false,
|
task: 'transcribe',
|
||||||
chunk_length_s: 3, // Shorter chunks for fallback
|
return_timestamps: false,
|
||||||
stride_length_s: 1, // Smaller stride for fallback
|
chunk_length_s: 3, // Shorter chunks for fallback
|
||||||
no_speech_threshold: 0.1, // Very low threshold for fallback
|
stride_length_s: 1, // Smaller stride for fallback
|
||||||
logprob_threshold: -1.2, // Very sensitive for fallback
|
no_speech_threshold: 0.1, // Very low threshold for fallback
|
||||||
compression_ratio_threshold: 2.5 // Very permissive for fallback
|
logprob_threshold: -1.2, // Very sensitive for fallback
|
||||||
})
|
compression_ratio_threshold: 2.5 // Very permissive for fallback
|
||||||
|
})
|
||||||
|
|
||||||
const fallbackText = fallbackResult?.text || ''
|
const fallbackText = fallbackResult?.text || ''
|
||||||
if (fallbackText.trim()) {
|
if (fallbackText.trim()) {
|
||||||
console.log(`✅ Fallback real-time transcript: "${fallbackText.trim()}"`)
|
console.log(`✅ Fallback real-time transcript: "${fallbackText.trim()}"`)
|
||||||
lastTranscriptionTimeRef.current = Date.now()
|
lastTranscriptionTimeRef.current = Date.now()
|
||||||
handleStreamingTranscriptUpdate(fallbackText.trim())
|
handleStreamingTranscriptUpdate(fallbackText.trim())
|
||||||
} else {
|
} else {
|
||||||
console.log('⚠️ Fallback transcription also produced no text')
|
console.log('⚠️ Fallback transcription also produced no text')
|
||||||
|
}
|
||||||
|
} catch (fallbackError) {
|
||||||
|
console.log('⚠️ Fallback transcription failed:', fallbackError)
|
||||||
}
|
}
|
||||||
} catch (fallbackError) {
|
|
||||||
console.log('⚠️ Fallback transcription failed:', fallbackError)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('❌ Error processing accumulated audio chunks:', error)
|
console.error('❌ Error processing accumulated audio chunks:', error)
|
||||||
}
|
}
|
||||||
}, [handleStreamingTranscriptUpdate, language])
|
}, [handleStreamingTranscriptUpdate, language, shouldUseRunPod])
|
||||||
|
|
||||||
// Process recorded audio chunks (final processing)
|
// Process recorded audio chunks (final processing)
|
||||||
const processAudioChunks = useCallback(async () => {
|
const processAudioChunks = useCallback(async () => {
|
||||||
if (!transcriberRef.current || audioChunksRef.current.length === 0) {
|
if (audioChunksRef.current.length === 0) {
|
||||||
console.log('⚠️ No transcriber or audio chunks to process')
|
console.log('⚠️ No audio chunks to process')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ensure model is loaded
|
// For local model, ensure transcriber is loaded
|
||||||
if (!modelLoaded) {
|
if (!shouldUseRunPod) {
|
||||||
console.log('⚠️ Model not loaded yet, waiting...')
|
if (!transcriberRef.current) {
|
||||||
try {
|
console.log('⚠️ No transcriber available')
|
||||||
await initializeTranscriber()
|
|
||||||
} catch (error) {
|
|
||||||
console.error('❌ Failed to initialize transcriber:', error)
|
|
||||||
onError?.(error as Error)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Ensure model is loaded
|
||||||
|
if (!modelLoaded) {
|
||||||
|
console.log('⚠️ Model not loaded yet, waiting...')
|
||||||
|
try {
|
||||||
|
await initializeTranscriber()
|
||||||
|
} catch (error) {
|
||||||
|
console.error('❌ Failed to initialize transcriber:', error)
|
||||||
|
onError?.(error as Error)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
@ -588,24 +663,32 @@ export const useWhisperTranscription = ({
|
||||||
|
|
||||||
console.log(`🎵 Processing audio: ${processedAudioData.length} samples (${(processedAudioData.length / 16000).toFixed(2)}s)`)
|
console.log(`🎵 Processing audio: ${processedAudioData.length} samples (${(processedAudioData.length / 16000).toFixed(2)}s)`)
|
||||||
|
|
||||||
// Check if transcriber is available
|
console.log('🔄 Starting transcription...')
|
||||||
if (!transcriberRef.current) {
|
|
||||||
console.error('❌ Transcriber not available for processing')
|
let newText = ''
|
||||||
throw new Error('Transcriber not initialized')
|
|
||||||
|
// Use RunPod if configured, otherwise use local model
|
||||||
|
if (shouldUseRunPod) {
|
||||||
|
console.log('🚀 Using RunPod WhisperX API...')
|
||||||
|
// Convert processed audio data back to blob for RunPod
|
||||||
|
// Create a WAV blob from the Float32Array
|
||||||
|
const wavBlob = await createWavBlob(processedAudioData, 16000)
|
||||||
|
newText = await transcribeWithRunPod(wavBlob, language)
|
||||||
|
console.log('✅ RunPod transcription result:', newText)
|
||||||
|
} else {
|
||||||
|
// Use local Whisper model
|
||||||
|
if (!transcriberRef.current) {
|
||||||
|
throw new Error('Transcriber not initialized')
|
||||||
|
}
|
||||||
|
const result = await transcriberRef.current(processedAudioData, {
|
||||||
|
language: language,
|
||||||
|
task: 'transcribe',
|
||||||
|
return_timestamps: false
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log('🔍 Transcription result:', result)
|
||||||
|
newText = result?.text?.trim() || ''
|
||||||
}
|
}
|
||||||
|
|
||||||
console.log('🔄 Starting transcription with Whisper model...')
|
|
||||||
|
|
||||||
// Transcribe the audio
|
|
||||||
const result = await transcriberRef.current(processedAudioData, {
|
|
||||||
language: language,
|
|
||||||
task: 'transcribe',
|
|
||||||
return_timestamps: false
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log('🔍 Transcription result:', result)
|
|
||||||
|
|
||||||
const newText = result?.text?.trim() || ''
|
|
||||||
if (newText) {
|
if (newText) {
|
||||||
const processedText = processTranscript(newText, enableStreaming)
|
const processedText = processTranscript(newText, enableStreaming)
|
||||||
|
|
||||||
|
|
@ -633,16 +716,17 @@ export const useWhisperTranscription = ({
|
||||||
console.log('⚠️ No transcription text produced')
|
console.log('⚠️ No transcription text produced')
|
||||||
console.log('🔍 Full transcription result object:', result)
|
console.log('🔍 Full transcription result object:', result)
|
||||||
|
|
||||||
// Try alternative transcription parameters
|
// Try alternative transcription parameters (only for local model)
|
||||||
console.log('🔄 Trying alternative transcription parameters...')
|
if (!shouldUseRunPod && transcriberRef.current) {
|
||||||
try {
|
console.log('🔄 Trying alternative transcription parameters...')
|
||||||
const altResult = await transcriberRef.current(processedAudioData, {
|
try {
|
||||||
task: 'transcribe',
|
const altResult = await transcriberRef.current(processedAudioData, {
|
||||||
return_timestamps: false
|
task: 'transcribe',
|
||||||
})
|
return_timestamps: false
|
||||||
console.log('🔍 Alternative transcription result:', altResult)
|
})
|
||||||
|
console.log('🔍 Alternative transcription result:', altResult)
|
||||||
|
|
||||||
if (altResult?.text?.trim()) {
|
if (altResult?.text?.trim()) {
|
||||||
const processedAltText = processTranscript(altResult.text, enableStreaming)
|
const processedAltText = processTranscript(altResult.text, enableStreaming)
|
||||||
console.log('✅ Alternative transcription successful:', processedAltText)
|
console.log('✅ Alternative transcription successful:', processedAltText)
|
||||||
const currentTranscript = transcriptRef.current
|
const currentTranscript = transcriptRef.current
|
||||||
|
|
@ -658,8 +742,9 @@ export const useWhisperTranscription = ({
|
||||||
previousTranscriptLengthRef.current = updatedTranscript.length
|
previousTranscriptLengthRef.current = updatedTranscript.length
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (altError) {
|
} catch (altError) {
|
||||||
console.log('⚠️ Alternative transcription also failed:', altError)
|
console.log('⚠️ Alternative transcription also failed:', altError)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -672,7 +757,7 @@ export const useWhisperTranscription = ({
|
||||||
} finally {
|
} finally {
|
||||||
setIsTranscribing(false)
|
setIsTranscribing(false)
|
||||||
}
|
}
|
||||||
}, [transcriberRef, language, onTranscriptUpdate, onError, enableStreaming, handleStreamingTranscriptUpdate, modelLoaded, initializeTranscriber])
|
}, [transcriberRef, language, onTranscriptUpdate, onError, enableStreaming, handleStreamingTranscriptUpdate, modelLoaded, initializeTranscriber, shouldUseRunPod])
|
||||||
|
|
||||||
// Start recording
|
// Start recording
|
||||||
const startRecording = useCallback(async () => {
|
const startRecording = useCallback(async () => {
|
||||||
|
|
@ -680,10 +765,13 @@ export const useWhisperTranscription = ({
|
||||||
console.log('🎤 Starting recording...')
|
console.log('🎤 Starting recording...')
|
||||||
console.log('🔍 enableStreaming in startRecording:', enableStreaming)
|
console.log('🔍 enableStreaming in startRecording:', enableStreaming)
|
||||||
|
|
||||||
// Ensure model is loaded before starting
|
// Ensure model is loaded before starting (skip for RunPod)
|
||||||
if (!modelLoaded) {
|
if (!shouldUseRunPod && !modelLoaded) {
|
||||||
console.log('🔄 Model not loaded, initializing...')
|
console.log('🔄 Model not loaded, initializing...')
|
||||||
await initializeTranscriber()
|
await initializeTranscriber()
|
||||||
|
} else if (shouldUseRunPod) {
|
||||||
|
// For RunPod, just mark as ready
|
||||||
|
setModelLoaded(true)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Don't reset transcripts for continuous transcription - keep existing content
|
// Don't reset transcripts for continuous transcription - keep existing content
|
||||||
|
|
@ -803,7 +891,7 @@ export const useWhisperTranscription = ({
|
||||||
console.error('❌ Error starting recording:', error)
|
console.error('❌ Error starting recording:', error)
|
||||||
onError?.(error as Error)
|
onError?.(error as Error)
|
||||||
}
|
}
|
||||||
}, [processAudioChunks, processAccumulatedAudioChunks, onError, enableStreaming, modelLoaded, initializeTranscriber])
|
}, [processAudioChunks, processAccumulatedAudioChunks, onError, enableStreaming, modelLoaded, initializeTranscriber, shouldUseRunPod])
|
||||||
|
|
||||||
// Stop recording
|
// Stop recording
|
||||||
const stopRecording = useCallback(async () => {
|
const stopRecording = useCallback(async () => {
|
||||||
|
|
@ -892,9 +980,11 @@ export const useWhisperTranscription = ({
|
||||||
periodicTranscriptionRef.current = null
|
periodicTranscriptionRef.current = null
|
||||||
}
|
}
|
||||||
|
|
||||||
// Initialize the model if not already loaded
|
// Initialize the model if not already loaded (skip for RunPod)
|
||||||
if (!modelLoaded) {
|
if (!shouldUseRunPod && !modelLoaded) {
|
||||||
await initializeTranscriber()
|
await initializeTranscriber()
|
||||||
|
} else if (shouldUseRunPod) {
|
||||||
|
setModelLoaded(true)
|
||||||
}
|
}
|
||||||
|
|
||||||
await startRecording()
|
await startRecording()
|
||||||
|
|
@ -933,7 +1023,7 @@ export const useWhisperTranscription = ({
|
||||||
if (autoInitialize) {
|
if (autoInitialize) {
|
||||||
initializeTranscriber().catch(console.warn)
|
initializeTranscriber().catch(console.warn)
|
||||||
}
|
}
|
||||||
}, [initializeTranscriber, autoInitialize])
|
}, [initializeTranscriber, autoInitialize, shouldUseRunPod])
|
||||||
|
|
||||||
// Cleanup on unmount
|
// Cleanup on unmount
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,8 @@ export interface ClientConfig {
|
||||||
webhookUrl?: string
|
webhookUrl?: string
|
||||||
webhookSecret?: string
|
webhookSecret?: string
|
||||||
openaiApiKey?: string
|
openaiApiKey?: string
|
||||||
|
runpodApiKey?: string
|
||||||
|
runpodEndpointId?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -38,6 +40,8 @@ export function getClientConfig(): ClientConfig {
|
||||||
webhookUrl: import.meta.env.VITE_QUARTZ_WEBHOOK_URL || import.meta.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
webhookUrl: import.meta.env.VITE_QUARTZ_WEBHOOK_URL || import.meta.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
||||||
webhookSecret: import.meta.env.VITE_QUARTZ_WEBHOOK_SECRET || import.meta.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
webhookSecret: import.meta.env.VITE_QUARTZ_WEBHOOK_SECRET || import.meta.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
||||||
openaiApiKey: import.meta.env.VITE_OPENAI_API_KEY || import.meta.env.NEXT_PUBLIC_OPENAI_API_KEY,
|
openaiApiKey: import.meta.env.VITE_OPENAI_API_KEY || import.meta.env.NEXT_PUBLIC_OPENAI_API_KEY,
|
||||||
|
runpodApiKey: import.meta.env.VITE_RUNPOD_API_KEY || import.meta.env.NEXT_PUBLIC_RUNPOD_API_KEY,
|
||||||
|
runpodEndpointId: import.meta.env.VITE_RUNPOD_ENDPOINT_ID || import.meta.env.NEXT_PUBLIC_RUNPOD_ENDPOINT_ID,
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
// Next.js environment
|
// Next.js environment
|
||||||
|
|
@ -52,6 +56,8 @@ export function getClientConfig(): ClientConfig {
|
||||||
webhookUrl: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
webhookUrl: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
||||||
webhookSecret: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
webhookSecret: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
||||||
openaiApiKey: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_OPENAI_API_KEY,
|
openaiApiKey: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_OPENAI_API_KEY,
|
||||||
|
runpodApiKey: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_RUNPOD_API_KEY,
|
||||||
|
runpodEndpointId: (window as any).__NEXT_DATA__?.env?.NEXT_PUBLIC_RUNPOD_ENDPOINT_ID,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
|
@ -66,10 +72,36 @@ export function getClientConfig(): ClientConfig {
|
||||||
quartzApiKey: process.env.VITE_QUARTZ_API_KEY || process.env.NEXT_PUBLIC_QUARTZ_API_KEY,
|
quartzApiKey: process.env.VITE_QUARTZ_API_KEY || process.env.NEXT_PUBLIC_QUARTZ_API_KEY,
|
||||||
webhookUrl: process.env.VITE_QUARTZ_WEBHOOK_URL || process.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
webhookUrl: process.env.VITE_QUARTZ_WEBHOOK_URL || process.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_URL,
|
||||||
webhookSecret: process.env.VITE_QUARTZ_WEBHOOK_SECRET || process.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
webhookSecret: process.env.VITE_QUARTZ_WEBHOOK_SECRET || process.env.NEXT_PUBLIC_QUARTZ_WEBHOOK_SECRET,
|
||||||
|
runpodApiKey: process.env.VITE_RUNPOD_API_KEY || process.env.NEXT_PUBLIC_RUNPOD_API_KEY,
|
||||||
|
runpodEndpointId: process.env.VITE_RUNPOD_ENDPOINT_ID || process.env.NEXT_PUBLIC_RUNPOD_ENDPOINT_ID,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get RunPod configuration for API calls
|
||||||
|
*/
|
||||||
|
export function getRunPodConfig(): { apiKey: string; endpointId: string } | null {
|
||||||
|
const config = getClientConfig()
|
||||||
|
|
||||||
|
if (!config.runpodApiKey || !config.runpodEndpointId) {
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
apiKey: config.runpodApiKey,
|
||||||
|
endpointId: config.runpodEndpointId
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if RunPod integration is configured
|
||||||
|
*/
|
||||||
|
export function isRunPodConfigured(): boolean {
|
||||||
|
const config = getClientConfig()
|
||||||
|
return !!(config.runpodApiKey && config.runpodEndpointId)
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if GitHub integration is configured
|
* Check if GitHub integration is configured
|
||||||
*/
|
*/
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,246 @@
|
||||||
|
/**
|
||||||
|
* RunPod API utility functions
|
||||||
|
* Handles communication with RunPod WhisperX endpoints
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { getRunPodConfig } from './clientConfig'
|
||||||
|
|
||||||
|
export interface RunPodTranscriptionResponse {
|
||||||
|
id?: string
|
||||||
|
status?: string
|
||||||
|
output?: {
|
||||||
|
text?: string
|
||||||
|
segments?: Array<{
|
||||||
|
start: number
|
||||||
|
end: number
|
||||||
|
text: string
|
||||||
|
}>
|
||||||
|
}
|
||||||
|
error?: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert audio blob to base64 string
|
||||||
|
*/
|
||||||
|
export async function blobToBase64(blob: Blob): Promise<string> {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const reader = new FileReader()
|
||||||
|
reader.onloadend = () => {
|
||||||
|
if (typeof reader.result === 'string') {
|
||||||
|
// Remove data URL prefix (e.g., "data:audio/webm;base64,")
|
||||||
|
const base64 = reader.result.split(',')[1] || reader.result
|
||||||
|
resolve(base64)
|
||||||
|
} else {
|
||||||
|
reject(new Error('Failed to convert blob to base64'))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reader.onerror = reject
|
||||||
|
reader.readAsDataURL(blob)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Send transcription request to RunPod endpoint
|
||||||
|
* Handles both synchronous and asynchronous job patterns
|
||||||
|
*/
|
||||||
|
export async function transcribeWithRunPod(
|
||||||
|
audioBlob: Blob,
|
||||||
|
language?: string
|
||||||
|
): Promise<string> {
|
||||||
|
const config = getRunPodConfig()
|
||||||
|
|
||||||
|
if (!config) {
|
||||||
|
throw new Error('RunPod API key or endpoint ID not configured. Please set VITE_RUNPOD_API_KEY and VITE_RUNPOD_ENDPOINT_ID environment variables.')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check audio blob size (limit to ~10MB to prevent issues)
|
||||||
|
const maxSize = 10 * 1024 * 1024 // 10MB
|
||||||
|
if (audioBlob.size > maxSize) {
|
||||||
|
throw new Error(`Audio file too large: ${(audioBlob.size / 1024 / 1024).toFixed(2)}MB. Maximum size is ${(maxSize / 1024 / 1024).toFixed(2)}MB`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert audio blob to base64
|
||||||
|
const audioBase64 = await blobToBase64(audioBlob)
|
||||||
|
|
||||||
|
// Detect audio format from blob type
|
||||||
|
const audioFormat = audioBlob.type || 'audio/wav'
|
||||||
|
|
||||||
|
const url = `https://api.runpod.ai/v2/${config.endpointId}/run`
|
||||||
|
|
||||||
|
// Prepare the request payload
|
||||||
|
// WhisperX typically expects audio as base64 or file URL
|
||||||
|
// The exact format may vary based on your WhisperX endpoint implementation
|
||||||
|
const requestBody = {
|
||||||
|
input: {
|
||||||
|
audio: audioBase64,
|
||||||
|
audio_format: audioFormat,
|
||||||
|
language: language || 'en',
|
||||||
|
task: 'transcribe'
|
||||||
|
// Note: Some WhisperX endpoints may expect different field names
|
||||||
|
// Adjust the requestBody structure in this function if needed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Add timeout to prevent hanging requests (30 seconds for initial request)
|
||||||
|
const controller = new AbortController()
|
||||||
|
const timeoutId = setTimeout(() => controller.abort(), 30000)
|
||||||
|
|
||||||
|
const response = await fetch(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Authorization': `Bearer ${config.apiKey}`
|
||||||
|
},
|
||||||
|
body: JSON.stringify(requestBody),
|
||||||
|
signal: controller.signal
|
||||||
|
})
|
||||||
|
|
||||||
|
clearTimeout(timeoutId)
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text()
|
||||||
|
console.error('RunPod API error response:', {
|
||||||
|
status: response.status,
|
||||||
|
statusText: response.statusText,
|
||||||
|
body: errorText
|
||||||
|
})
|
||||||
|
throw new Error(`RunPod API error: ${response.status} - ${errorText}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const data: RunPodTranscriptionResponse = await response.json()
|
||||||
|
|
||||||
|
console.log('RunPod initial response:', data)
|
||||||
|
|
||||||
|
// Handle async job pattern (RunPod often returns job IDs)
|
||||||
|
if (data.id && (data.status === 'IN_QUEUE' || data.status === 'IN_PROGRESS')) {
|
||||||
|
console.log('Job is async, polling for results...', data.id)
|
||||||
|
return await pollRunPodJob(data.id, config.apiKey, config.endpointId)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle direct response
|
||||||
|
if (data.output?.text) {
|
||||||
|
return data.output.text.trim()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle error response
|
||||||
|
if (data.error) {
|
||||||
|
throw new Error(`RunPod transcription error: ${data.error}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback: try to extract text from segments
|
||||||
|
if (data.output?.segments && data.output.segments.length > 0) {
|
||||||
|
return data.output.segments.map(seg => seg.text).join(' ').trim()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if response has unexpected structure
|
||||||
|
console.warn('Unexpected RunPod response structure:', data)
|
||||||
|
throw new Error('No transcription text found in RunPod response. Check endpoint response format.')
|
||||||
|
} catch (error: any) {
|
||||||
|
if (error.name === 'AbortError') {
|
||||||
|
throw new Error('RunPod request timed out after 30 seconds')
|
||||||
|
}
|
||||||
|
console.error('RunPod transcription error:', error)
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Poll RunPod job status until completion
|
||||||
|
*/
|
||||||
|
async function pollRunPodJob(
|
||||||
|
jobId: string,
|
||||||
|
apiKey: string,
|
||||||
|
endpointId: string,
|
||||||
|
maxAttempts: number = 120, // Increased to 120 attempts (2 minutes at 1s intervals)
|
||||||
|
pollInterval: number = 1000
|
||||||
|
): Promise<string> {
|
||||||
|
const statusUrl = `https://api.runpod.ai/v2/${endpointId}/status/${jobId}`
|
||||||
|
|
||||||
|
console.log(`Polling job ${jobId} (max ${maxAttempts} attempts, ${pollInterval}ms interval)`)
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
||||||
|
try {
|
||||||
|
// Add timeout for each status check (5 seconds)
|
||||||
|
const controller = new AbortController()
|
||||||
|
const timeoutId = setTimeout(() => controller.abort(), 5000)
|
||||||
|
|
||||||
|
const response = await fetch(statusUrl, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
},
|
||||||
|
signal: controller.signal
|
||||||
|
})
|
||||||
|
|
||||||
|
clearTimeout(timeoutId)
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text()
|
||||||
|
console.error(`Job status check failed (attempt ${attempt + 1}/${maxAttempts}):`, {
|
||||||
|
status: response.status,
|
||||||
|
statusText: response.statusText,
|
||||||
|
body: errorText
|
||||||
|
})
|
||||||
|
|
||||||
|
// Don't fail immediately on 404 - job might still be processing
|
||||||
|
if (response.status === 404 && attempt < maxAttempts - 1) {
|
||||||
|
console.log('Job not found yet, continuing to poll...')
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`Failed to check job status: ${response.status} - ${errorText}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const data: RunPodTranscriptionResponse = await response.json()
|
||||||
|
|
||||||
|
console.log(`Job status (attempt ${attempt + 1}/${maxAttempts}):`, data.status)
|
||||||
|
|
||||||
|
if (data.status === 'COMPLETED') {
|
||||||
|
console.log('Job completed, extracting transcription...')
|
||||||
|
|
||||||
|
if (data.output?.text) {
|
||||||
|
return data.output.text.trim()
|
||||||
|
}
|
||||||
|
if (data.output?.segments && data.output.segments.length > 0) {
|
||||||
|
return data.output.segments.map(seg => seg.text).join(' ').trim()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log the full response for debugging
|
||||||
|
console.error('Job completed but no transcription found. Full response:', JSON.stringify(data, null, 2))
|
||||||
|
throw new Error('Job completed but no transcription text found in response')
|
||||||
|
}
|
||||||
|
|
||||||
|
if (data.status === 'FAILED') {
|
||||||
|
const errorMsg = data.error || 'Unknown error'
|
||||||
|
console.error('Job failed:', errorMsg)
|
||||||
|
throw new Error(`Job failed: ${errorMsg}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Job still in progress, wait and retry
|
||||||
|
if (attempt % 10 === 0) {
|
||||||
|
console.log(`Job still processing... (${attempt + 1}/${maxAttempts} attempts)`)
|
||||||
|
}
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
} catch (error: any) {
|
||||||
|
if (error.name === 'AbortError') {
|
||||||
|
console.warn(`Status check timed out (attempt ${attempt + 1}/${maxAttempts})`)
|
||||||
|
if (attempt < maxAttempts - 1) {
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
throw new Error('Status check timed out multiple times')
|
||||||
|
}
|
||||||
|
|
||||||
|
if (attempt === maxAttempts - 1) {
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
// Wait before retrying
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`Job polling timeout after ${maxAttempts} attempts (${(maxAttempts * pollInterval / 1000).toFixed(0)} seconds)`)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
@ -42,6 +42,8 @@ import { HolonBrowserShape } from "@/shapes/HolonBrowserShapeUtil"
|
||||||
import { ObsidianBrowserShape } from "@/shapes/ObsidianBrowserShapeUtil"
|
import { ObsidianBrowserShape } from "@/shapes/ObsidianBrowserShapeUtil"
|
||||||
import { FathomMeetingsBrowserShape } from "@/shapes/FathomMeetingsBrowserShapeUtil"
|
import { FathomMeetingsBrowserShape } from "@/shapes/FathomMeetingsBrowserShapeUtil"
|
||||||
import { LocationShareShape } from "@/shapes/LocationShareShapeUtil"
|
import { LocationShareShape } from "@/shapes/LocationShareShapeUtil"
|
||||||
|
import { ImageGenShape } from "@/shapes/ImageGenShapeUtil"
|
||||||
|
import { ImageGenTool } from "@/tools/ImageGenTool"
|
||||||
import {
|
import {
|
||||||
lockElement,
|
lockElement,
|
||||||
unlockElement,
|
unlockElement,
|
||||||
|
|
@ -82,6 +84,7 @@ const customShapeUtils = [
|
||||||
ObsidianBrowserShape,
|
ObsidianBrowserShape,
|
||||||
FathomMeetingsBrowserShape,
|
FathomMeetingsBrowserShape,
|
||||||
LocationShareShape,
|
LocationShareShape,
|
||||||
|
ImageGenShape,
|
||||||
]
|
]
|
||||||
const customTools = [
|
const customTools = [
|
||||||
ChatBoxTool,
|
ChatBoxTool,
|
||||||
|
|
@ -96,6 +99,7 @@ const customTools = [
|
||||||
TranscriptionTool,
|
TranscriptionTool,
|
||||||
HolonTool,
|
HolonTool,
|
||||||
FathomMeetingsTool,
|
FathomMeetingsTool,
|
||||||
|
ImageGenTool,
|
||||||
]
|
]
|
||||||
|
|
||||||
export function Board() {
|
export function Board() {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,730 @@
|
||||||
|
import {
|
||||||
|
BaseBoxShapeUtil,
|
||||||
|
Geometry2d,
|
||||||
|
HTMLContainer,
|
||||||
|
Rectangle2d,
|
||||||
|
TLBaseShape,
|
||||||
|
} from "tldraw"
|
||||||
|
import React, { useState } from "react"
|
||||||
|
import { getRunPodConfig } from "@/lib/clientConfig"
|
||||||
|
|
||||||
|
// Feature flag: Set to false when RunPod API is ready for production
|
||||||
|
const USE_MOCK_API = true
|
||||||
|
|
||||||
|
// Type definition for RunPod API responses
|
||||||
|
interface RunPodJobResponse {
|
||||||
|
id?: string
|
||||||
|
status?: 'IN_QUEUE' | 'IN_PROGRESS' | 'STARTING' | 'COMPLETED' | 'FAILED' | 'CANCELLED'
|
||||||
|
output?: string | {
|
||||||
|
image?: string
|
||||||
|
url?: string
|
||||||
|
images?: Array<{ data?: string; url?: string; filename?: string; type?: string }>
|
||||||
|
result?: string
|
||||||
|
[key: string]: any
|
||||||
|
}
|
||||||
|
error?: string
|
||||||
|
image?: string
|
||||||
|
url?: string
|
||||||
|
result?: string | {
|
||||||
|
image?: string
|
||||||
|
url?: string
|
||||||
|
[key: string]: any
|
||||||
|
}
|
||||||
|
[key: string]: any
|
||||||
|
}
|
||||||
|
|
||||||
|
type IImageGen = TLBaseShape<
|
||||||
|
"ImageGen",
|
||||||
|
{
|
||||||
|
w: number
|
||||||
|
h: number
|
||||||
|
prompt: string
|
||||||
|
imageUrl: string | null
|
||||||
|
isLoading: boolean
|
||||||
|
error: string | null
|
||||||
|
endpointId?: string // Optional custom endpoint ID
|
||||||
|
}
|
||||||
|
>
|
||||||
|
|
||||||
|
// Helper function to poll RunPod job status until completion
|
||||||
|
async function pollRunPodJob(
|
||||||
|
jobId: string,
|
||||||
|
apiKey: string,
|
||||||
|
endpointId: string,
|
||||||
|
maxAttempts: number = 60,
|
||||||
|
pollInterval: number = 2000
|
||||||
|
): Promise<string> {
|
||||||
|
const statusUrl = `https://api.runpod.ai/v2/${endpointId}/status/${jobId}`
|
||||||
|
console.log('🔄 ImageGen: Polling job:', jobId)
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
||||||
|
try {
|
||||||
|
const response = await fetch(statusUrl, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text()
|
||||||
|
console.error(`❌ ImageGen: Poll error (attempt ${attempt + 1}/${maxAttempts}):`, response.status, errorText)
|
||||||
|
throw new Error(`Failed to check job status: ${response.status} - ${errorText}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json() as RunPodJobResponse
|
||||||
|
console.log(`🔄 ImageGen: Poll attempt ${attempt + 1}/${maxAttempts}, status:`, data.status)
|
||||||
|
console.log(`📋 ImageGen: Full response data:`, JSON.stringify(data, null, 2))
|
||||||
|
|
||||||
|
if (data.status === 'COMPLETED') {
|
||||||
|
console.log('✅ ImageGen: Job completed, processing output...')
|
||||||
|
|
||||||
|
// Extract image URL from various possible response formats
|
||||||
|
let imageUrl = ''
|
||||||
|
|
||||||
|
// Check if output exists at all
|
||||||
|
if (!data.output) {
|
||||||
|
// Only retry 2-3 times, then proceed to check alternatives
|
||||||
|
if (attempt < 3) {
|
||||||
|
console.log(`⏳ ImageGen: COMPLETED but no output yet, waiting briefly (attempt ${attempt + 1}/3)...`)
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 500))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try alternative ways to get the output - maybe it's at the top level
|
||||||
|
console.log('⚠️ ImageGen: No output field found, checking for alternative response formats...')
|
||||||
|
console.log('📋 ImageGen: All available fields:', Object.keys(data))
|
||||||
|
|
||||||
|
// Check if image data is at top level
|
||||||
|
if (data.image) {
|
||||||
|
imageUrl = data.image
|
||||||
|
console.log('✅ ImageGen: Found image at top level')
|
||||||
|
} else if (data.url) {
|
||||||
|
imageUrl = data.url
|
||||||
|
console.log('✅ ImageGen: Found url at top level')
|
||||||
|
} else if (data.result) {
|
||||||
|
// Some endpoints return result instead of output
|
||||||
|
if (typeof data.result === 'string') {
|
||||||
|
imageUrl = data.result
|
||||||
|
} else if (data.result.image) {
|
||||||
|
imageUrl = data.result.image
|
||||||
|
} else if (data.result.url) {
|
||||||
|
imageUrl = data.result.url
|
||||||
|
}
|
||||||
|
console.log('✅ ImageGen: Found result field')
|
||||||
|
} else {
|
||||||
|
// Last resort: try to fetch output via stream endpoint (some RunPod endpoints use this)
|
||||||
|
console.log('⚠️ ImageGen: Trying alternative endpoint to retrieve output...')
|
||||||
|
try {
|
||||||
|
const streamUrl = `https://api.runpod.ai/v2/${endpointId}/stream/${jobId}`
|
||||||
|
const streamResponse = await fetch(streamUrl, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
if (streamResponse.ok) {
|
||||||
|
const streamData = await streamResponse.json() as RunPodJobResponse
|
||||||
|
console.log('📥 ImageGen: Stream endpoint response:', JSON.stringify(streamData, null, 2))
|
||||||
|
|
||||||
|
if (streamData.output) {
|
||||||
|
if (typeof streamData.output === 'string') {
|
||||||
|
imageUrl = streamData.output
|
||||||
|
} else if (streamData.output.image) {
|
||||||
|
imageUrl = streamData.output.image
|
||||||
|
} else if (streamData.output.url) {
|
||||||
|
imageUrl = streamData.output.url
|
||||||
|
} else if (Array.isArray(streamData.output.images) && streamData.output.images.length > 0) {
|
||||||
|
const firstImage = streamData.output.images[0]
|
||||||
|
if (firstImage.data) {
|
||||||
|
imageUrl = firstImage.data.startsWith('data:') ? firstImage.data : `data:image/${firstImage.type || 'png'};base64,${firstImage.data}`
|
||||||
|
} else if (firstImage.url) {
|
||||||
|
imageUrl = firstImage.url
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (imageUrl) {
|
||||||
|
console.log('✅ ImageGen: Found image URL via stream endpoint')
|
||||||
|
return imageUrl
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (streamError) {
|
||||||
|
console.log('⚠️ ImageGen: Stream endpoint not available or failed:', streamError)
|
||||||
|
}
|
||||||
|
|
||||||
|
console.error('❌ ImageGen: Job completed but no output field in response after retries:', JSON.stringify(data, null, 2))
|
||||||
|
throw new Error(
|
||||||
|
'Job completed but no output data found.\n\n' +
|
||||||
|
'Possible issues:\n' +
|
||||||
|
'1. The RunPod endpoint handler may not be returning output correctly\n' +
|
||||||
|
'2. Check the endpoint handler logs in RunPod console\n' +
|
||||||
|
'3. Verify the handler returns: { output: { image: "url" } } or { output: "url" }\n' +
|
||||||
|
'4. For ComfyUI workers, ensure output.images array is returned\n' +
|
||||||
|
'5. The endpoint may need to be reconfigured\n\n' +
|
||||||
|
'Response received: ' + JSON.stringify(data, null, 2)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Extract image URL from various possible response formats
|
||||||
|
if (typeof data.output === 'string') {
|
||||||
|
imageUrl = data.output
|
||||||
|
} else if (data.output?.image) {
|
||||||
|
imageUrl = data.output.image
|
||||||
|
} else if (data.output?.url) {
|
||||||
|
imageUrl = data.output.url
|
||||||
|
} else if (data.output?.output) {
|
||||||
|
// Handle nested output structure
|
||||||
|
if (typeof data.output.output === 'string') {
|
||||||
|
imageUrl = data.output.output
|
||||||
|
} else if (data.output.output?.image) {
|
||||||
|
imageUrl = data.output.output.image
|
||||||
|
} else if (data.output.output?.url) {
|
||||||
|
imageUrl = data.output.output.url
|
||||||
|
}
|
||||||
|
} else if (Array.isArray(data.output) && data.output.length > 0) {
|
||||||
|
// Handle array responses
|
||||||
|
const firstItem = data.output[0]
|
||||||
|
if (typeof firstItem === 'string') {
|
||||||
|
imageUrl = firstItem
|
||||||
|
} else if (firstItem.image) {
|
||||||
|
imageUrl = firstItem.image
|
||||||
|
} else if (firstItem.url) {
|
||||||
|
imageUrl = firstItem.url
|
||||||
|
}
|
||||||
|
} else if (data.output?.result) {
|
||||||
|
// Some formats nest result inside output
|
||||||
|
if (typeof data.output.result === 'string') {
|
||||||
|
imageUrl = data.output.result
|
||||||
|
} else if (data.output.result?.image) {
|
||||||
|
imageUrl = data.output.result.image
|
||||||
|
} else if (data.output.result?.url) {
|
||||||
|
imageUrl = data.output.result.url
|
||||||
|
}
|
||||||
|
} else if (Array.isArray(data.output?.images) && data.output.images.length > 0) {
|
||||||
|
// ComfyUI worker format: { output: { images: [{ filename, type, data }] } }
|
||||||
|
const firstImage = data.output.images[0]
|
||||||
|
if (firstImage.data) {
|
||||||
|
// Base64 encoded image
|
||||||
|
if (firstImage.data.startsWith('data:image')) {
|
||||||
|
imageUrl = firstImage.data
|
||||||
|
} else if (firstImage.data.startsWith('http')) {
|
||||||
|
imageUrl = firstImage.data
|
||||||
|
} else {
|
||||||
|
// Assume base64 without prefix
|
||||||
|
imageUrl = `data:image/${firstImage.type || 'png'};base64,${firstImage.data}`
|
||||||
|
}
|
||||||
|
console.log('✅ ImageGen: Found image in ComfyUI format (images array)')
|
||||||
|
} else if (firstImage.url) {
|
||||||
|
imageUrl = firstImage.url
|
||||||
|
console.log('✅ ImageGen: Found image URL in ComfyUI format')
|
||||||
|
} else if (firstImage.filename) {
|
||||||
|
// Try to construct URL from filename (may need endpoint-specific handling)
|
||||||
|
console.log('⚠️ ImageGen: Found filename but no URL, filename:', firstImage.filename)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!imageUrl || imageUrl.trim() === '') {
|
||||||
|
console.error('❌ ImageGen: No image URL found in response:', JSON.stringify(data, null, 2))
|
||||||
|
throw new Error(
|
||||||
|
'Job completed but no image URL found in output.\n\n' +
|
||||||
|
'Expected formats:\n' +
|
||||||
|
'- { output: "https://..." }\n' +
|
||||||
|
'- { output: { image: "https://..." } }\n' +
|
||||||
|
'- { output: { url: "https://..." } }\n' +
|
||||||
|
'- { output: ["https://..."] }\n\n' +
|
||||||
|
'Received: ' + JSON.stringify(data, null, 2)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return imageUrl
|
||||||
|
}
|
||||||
|
|
||||||
|
if (data.status === 'FAILED') {
|
||||||
|
console.error('❌ ImageGen: Job failed:', data.error || 'Unknown error')
|
||||||
|
throw new Error(`Job failed: ${data.error || 'Unknown error'}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait before next poll
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
} catch (error) {
|
||||||
|
// If we get COMPLETED status without output, don't retry - fail immediately
|
||||||
|
const errorMessage = error instanceof Error ? error.message : String(error)
|
||||||
|
if (errorMessage.includes('no output') || errorMessage.includes('no image URL')) {
|
||||||
|
console.error('❌ ImageGen: Stopping polling due to missing output data')
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
|
||||||
|
// For other errors, retry up to maxAttempts
|
||||||
|
if (attempt === maxAttempts - 1) {
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error('Job polling timed out')
|
||||||
|
}
|
||||||
|
|
||||||
|
export class ImageGenShape extends BaseBoxShapeUtil<IImageGen> {
|
||||||
|
static override type = "ImageGen" as const
|
||||||
|
|
||||||
|
MIN_WIDTH = 300 as const
|
||||||
|
MIN_HEIGHT = 300 as const
|
||||||
|
DEFAULT_WIDTH = 400 as const
|
||||||
|
DEFAULT_HEIGHT = 400 as const
|
||||||
|
|
||||||
|
getDefaultProps(): IImageGen["props"] {
|
||||||
|
return {
|
||||||
|
w: this.DEFAULT_WIDTH,
|
||||||
|
h: this.DEFAULT_HEIGHT,
|
||||||
|
prompt: "",
|
||||||
|
imageUrl: null,
|
||||||
|
isLoading: false,
|
||||||
|
error: null,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
getGeometry(shape: IImageGen): Geometry2d {
|
||||||
|
return new Rectangle2d({
|
||||||
|
width: shape.props.w,
|
||||||
|
height: shape.props.h,
|
||||||
|
isFilled: true,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
component(shape: IImageGen) {
|
||||||
|
const [isHovering, setIsHovering] = useState(false)
|
||||||
|
const isSelected = this.editor.getSelectedShapeIds().includes(shape.id)
|
||||||
|
|
||||||
|
const generateImage = async (prompt: string) => {
|
||||||
|
console.log("🎨 ImageGen: Generating image with prompt:", prompt)
|
||||||
|
|
||||||
|
// Clear any previous errors
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
error: null,
|
||||||
|
isLoading: true,
|
||||||
|
imageUrl: null
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Get RunPod configuration
|
||||||
|
const runpodConfig = getRunPodConfig()
|
||||||
|
const endpointId = shape.props.endpointId || runpodConfig?.endpointId || "tzf1j3sc3zufsy"
|
||||||
|
const apiKey = runpodConfig?.apiKey
|
||||||
|
|
||||||
|
// Mock API mode: Return placeholder image without calling RunPod
|
||||||
|
if (USE_MOCK_API) {
|
||||||
|
console.log("🎭 ImageGen: Using MOCK API mode (no real RunPod call)")
|
||||||
|
console.log("🎨 ImageGen: Mock prompt:", prompt)
|
||||||
|
|
||||||
|
// Simulate API delay
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 1500))
|
||||||
|
|
||||||
|
// Use a placeholder image service
|
||||||
|
const mockImageUrl = `https://via.placeholder.com/512x512/4F46E5/FFFFFF?text=${encodeURIComponent(prompt.substring(0, 30))}`
|
||||||
|
|
||||||
|
console.log("✅ ImageGen: Mock image generated:", mockImageUrl)
|
||||||
|
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
imageUrl: mockImageUrl,
|
||||||
|
isLoading: false,
|
||||||
|
error: null
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Real API mode: Use RunPod
|
||||||
|
if (!apiKey) {
|
||||||
|
throw new Error("RunPod API key not configured. Please set VITE_RUNPOD_API_KEY environment variable.")
|
||||||
|
}
|
||||||
|
|
||||||
|
const url = `https://api.runpod.ai/v2/${endpointId}/run`
|
||||||
|
|
||||||
|
console.log("📤 ImageGen: Sending request to:", url)
|
||||||
|
|
||||||
|
const response = await fetch(url, {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Authorization": `Bearer ${apiKey}`
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
input: {
|
||||||
|
prompt: prompt
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text()
|
||||||
|
console.error("❌ ImageGen: Error response:", errorText)
|
||||||
|
throw new Error(`HTTP error! status: ${response.status} - ${errorText}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json() as RunPodJobResponse
|
||||||
|
console.log("📥 ImageGen: Response data:", JSON.stringify(data, null, 2))
|
||||||
|
|
||||||
|
// Handle async job pattern (RunPod often returns job IDs)
|
||||||
|
if (data.id && (data.status === 'IN_QUEUE' || data.status === 'IN_PROGRESS' || data.status === 'STARTING')) {
|
||||||
|
console.log("⏳ ImageGen: Job queued/in progress, polling job ID:", data.id)
|
||||||
|
const imageUrl = await pollRunPodJob(data.id, apiKey, endpointId)
|
||||||
|
console.log("✅ ImageGen: Job completed, image URL:", imageUrl)
|
||||||
|
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
imageUrl: imageUrl,
|
||||||
|
isLoading: false,
|
||||||
|
error: null
|
||||||
|
},
|
||||||
|
})
|
||||||
|
} else if (data.output) {
|
||||||
|
// Handle direct response
|
||||||
|
let imageUrl = ''
|
||||||
|
if (typeof data.output === 'string') {
|
||||||
|
imageUrl = data.output
|
||||||
|
} else if (data.output.image) {
|
||||||
|
imageUrl = data.output.image
|
||||||
|
} else if (data.output.url) {
|
||||||
|
imageUrl = data.output.url
|
||||||
|
} else if (Array.isArray(data.output) && data.output.length > 0) {
|
||||||
|
const firstItem = data.output[0]
|
||||||
|
if (typeof firstItem === 'string') {
|
||||||
|
imageUrl = firstItem
|
||||||
|
} else if (firstItem.image) {
|
||||||
|
imageUrl = firstItem.image
|
||||||
|
} else if (firstItem.url) {
|
||||||
|
imageUrl = firstItem.url
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (imageUrl) {
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
imageUrl: imageUrl,
|
||||||
|
isLoading: false,
|
||||||
|
error: null
|
||||||
|
},
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
throw new Error("No image URL found in response")
|
||||||
|
}
|
||||||
|
} else if (data.error) {
|
||||||
|
throw new Error(`RunPod API error: ${data.error}`)
|
||||||
|
} else {
|
||||||
|
throw new Error("No valid response from RunPod API")
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
const errorMessage = error instanceof Error ? error.message : String(error)
|
||||||
|
console.error("❌ ImageGen: Error:", errorMessage)
|
||||||
|
|
||||||
|
let userFriendlyError = ''
|
||||||
|
|
||||||
|
if (errorMessage.includes('API key not configured')) {
|
||||||
|
userFriendlyError = '❌ RunPod API key not configured. Please set VITE_RUNPOD_API_KEY environment variable.'
|
||||||
|
} else if (errorMessage.includes('401') || errorMessage.includes('403') || errorMessage.includes('Unauthorized')) {
|
||||||
|
userFriendlyError = '❌ API key authentication failed. Please check your RunPod API key.'
|
||||||
|
} else if (errorMessage.includes('404')) {
|
||||||
|
userFriendlyError = '❌ Endpoint not found. Please check your endpoint ID.'
|
||||||
|
} else if (errorMessage.includes('no output data found') || errorMessage.includes('no image URL found')) {
|
||||||
|
// For multi-line error messages, show a concise version in the UI
|
||||||
|
// The full details are already in the console
|
||||||
|
userFriendlyError = '❌ Image generation completed but no image data was returned.\n\n' +
|
||||||
|
'This usually means the RunPod endpoint handler is not configured correctly.\n\n' +
|
||||||
|
'Please check:\n' +
|
||||||
|
'1. RunPod endpoint handler logs\n' +
|
||||||
|
'2. Handler returns: { output: { image: "url" } }\n' +
|
||||||
|
'3. See browser console for full details'
|
||||||
|
} else {
|
||||||
|
// Truncate very long error messages for UI display
|
||||||
|
const maxLength = 500
|
||||||
|
if (errorMessage.length > maxLength) {
|
||||||
|
userFriendlyError = `❌ Error: ${errorMessage.substring(0, maxLength)}...\n\n(Full error in console)`
|
||||||
|
} else {
|
||||||
|
userFriendlyError = `❌ Error: ${errorMessage}`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
isLoading: false,
|
||||||
|
error: userFriendlyError
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleGenerate = () => {
|
||||||
|
if (shape.props.prompt.trim() && !shape.props.isLoading) {
|
||||||
|
generateImage(shape.props.prompt)
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: { prompt: "" },
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<HTMLContainer
|
||||||
|
style={{
|
||||||
|
borderRadius: 6,
|
||||||
|
border: "1px solid lightgrey",
|
||||||
|
padding: 8,
|
||||||
|
height: shape.props.h,
|
||||||
|
width: shape.props.w,
|
||||||
|
pointerEvents: isSelected || isHovering ? "all" : "none",
|
||||||
|
backgroundColor: "#ffffff",
|
||||||
|
overflow: "hidden",
|
||||||
|
display: "flex",
|
||||||
|
flexDirection: "column",
|
||||||
|
gap: 8,
|
||||||
|
}}
|
||||||
|
onPointerEnter={() => setIsHovering(true)}
|
||||||
|
onPointerLeave={() => setIsHovering(false)}
|
||||||
|
>
|
||||||
|
{/* Error Display */}
|
||||||
|
{shape.props.error && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
padding: "12px 16px",
|
||||||
|
backgroundColor: "#fee",
|
||||||
|
border: "1px solid #fcc",
|
||||||
|
borderRadius: "8px",
|
||||||
|
color: "#c33",
|
||||||
|
fontSize: "13px",
|
||||||
|
display: "flex",
|
||||||
|
alignItems: "flex-start",
|
||||||
|
gap: "8px",
|
||||||
|
whiteSpace: "pre-wrap",
|
||||||
|
wordBreak: "break-word",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<span style={{ fontSize: "18px", flexShrink: 0 }}>⚠️</span>
|
||||||
|
<span style={{ flex: 1, lineHeight: "1.5" }}>{shape.props.error}</span>
|
||||||
|
<button
|
||||||
|
onClick={() => {
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: { error: null },
|
||||||
|
})
|
||||||
|
}}
|
||||||
|
style={{
|
||||||
|
padding: "4px 8px",
|
||||||
|
backgroundColor: "#fcc",
|
||||||
|
border: "1px solid #c99",
|
||||||
|
borderRadius: "4px",
|
||||||
|
cursor: "pointer",
|
||||||
|
fontSize: "11px",
|
||||||
|
flexShrink: 0,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Dismiss
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Image Display */}
|
||||||
|
{shape.props.imageUrl && !shape.props.isLoading && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
display: "flex",
|
||||||
|
alignItems: "center",
|
||||||
|
justifyContent: "center",
|
||||||
|
backgroundColor: "#f5f5f5",
|
||||||
|
borderRadius: "4px",
|
||||||
|
overflow: "hidden",
|
||||||
|
minHeight: 0,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<img
|
||||||
|
src={shape.props.imageUrl}
|
||||||
|
alt={shape.props.prompt || "Generated image"}
|
||||||
|
style={{
|
||||||
|
maxWidth: "100%",
|
||||||
|
maxHeight: "100%",
|
||||||
|
objectFit: "contain",
|
||||||
|
}}
|
||||||
|
onError={(_e) => {
|
||||||
|
console.error("❌ ImageGen: Failed to load image:", shape.props.imageUrl)
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: {
|
||||||
|
error: "Failed to load generated image",
|
||||||
|
imageUrl: null
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Loading State */}
|
||||||
|
{shape.props.isLoading && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
display: "flex",
|
||||||
|
flexDirection: "column",
|
||||||
|
alignItems: "center",
|
||||||
|
justifyContent: "center",
|
||||||
|
backgroundColor: "#f5f5f5",
|
||||||
|
borderRadius: "4px",
|
||||||
|
gap: 12,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
width: 40,
|
||||||
|
height: 40,
|
||||||
|
border: "4px solid #f3f3f3",
|
||||||
|
borderTop: "4px solid #007AFF",
|
||||||
|
borderRadius: "50%",
|
||||||
|
animation: "spin 1s linear infinite",
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<span style={{ color: "#666", fontSize: "14px" }}>
|
||||||
|
Generating image...
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Empty State */}
|
||||||
|
{!shape.props.imageUrl && !shape.props.isLoading && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
display: "flex",
|
||||||
|
alignItems: "center",
|
||||||
|
justifyContent: "center",
|
||||||
|
backgroundColor: "#f5f5f5",
|
||||||
|
borderRadius: "4px",
|
||||||
|
color: "#999",
|
||||||
|
fontSize: "14px",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Generated image will appear here
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Input Section */}
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
display: "flex",
|
||||||
|
gap: 8,
|
||||||
|
pointerEvents: isSelected || isHovering ? "all" : "none",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<input
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
height: "36px",
|
||||||
|
backgroundColor: "rgba(0, 0, 0, 0.05)",
|
||||||
|
border: "1px solid rgba(0, 0, 0, 0.1)",
|
||||||
|
borderRadius: "4px",
|
||||||
|
fontSize: 14,
|
||||||
|
padding: "0 8px",
|
||||||
|
}}
|
||||||
|
type="text"
|
||||||
|
placeholder="Enter image prompt..."
|
||||||
|
value={shape.props.prompt}
|
||||||
|
onChange={(e) => {
|
||||||
|
this.editor.updateShape<IImageGen>({
|
||||||
|
id: shape.id,
|
||||||
|
type: "ImageGen",
|
||||||
|
props: { prompt: e.target.value },
|
||||||
|
})
|
||||||
|
}}
|
||||||
|
onKeyDown={(e) => {
|
||||||
|
e.stopPropagation()
|
||||||
|
if (e.key === 'Enter' && !e.shiftKey) {
|
||||||
|
e.preventDefault()
|
||||||
|
if (shape.props.prompt.trim() && !shape.props.isLoading) {
|
||||||
|
handleGenerate()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
onPointerDown={(e) => {
|
||||||
|
e.stopPropagation()
|
||||||
|
}}
|
||||||
|
onClick={(e) => {
|
||||||
|
e.stopPropagation()
|
||||||
|
}}
|
||||||
|
disabled={shape.props.isLoading}
|
||||||
|
/>
|
||||||
|
<button
|
||||||
|
style={{
|
||||||
|
height: "36px",
|
||||||
|
padding: "0 16px",
|
||||||
|
pointerEvents: "all",
|
||||||
|
cursor: shape.props.prompt.trim() && !shape.props.isLoading ? "pointer" : "not-allowed",
|
||||||
|
backgroundColor: shape.props.prompt.trim() && !shape.props.isLoading ? "#007AFF" : "#ccc",
|
||||||
|
color: "white",
|
||||||
|
border: "none",
|
||||||
|
borderRadius: "4px",
|
||||||
|
fontWeight: "500",
|
||||||
|
fontSize: "14px",
|
||||||
|
opacity: shape.props.prompt.trim() && !shape.props.isLoading ? 1 : 0.6,
|
||||||
|
}}
|
||||||
|
onPointerDown={(e) => {
|
||||||
|
e.stopPropagation()
|
||||||
|
e.preventDefault()
|
||||||
|
if (shape.props.prompt.trim() && !shape.props.isLoading) {
|
||||||
|
handleGenerate()
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
onClick={(e) => {
|
||||||
|
e.preventDefault()
|
||||||
|
e.stopPropagation()
|
||||||
|
if (shape.props.prompt.trim() && !shape.props.isLoading) {
|
||||||
|
handleGenerate()
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
disabled={shape.props.isLoading || !shape.props.prompt.trim()}
|
||||||
|
>
|
||||||
|
Generate
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Add CSS for spinner animation */}
|
||||||
|
<style>{`
|
||||||
|
@keyframes spin {
|
||||||
|
0% { transform: rotate(0deg); }
|
||||||
|
100% { transform: rotate(360deg); }
|
||||||
|
}
|
||||||
|
`}</style>
|
||||||
|
</HTMLContainer>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override indicator(shape: IImageGen) {
|
||||||
|
return (
|
||||||
|
<rect
|
||||||
|
width={shape.props.w}
|
||||||
|
height={shape.props.h}
|
||||||
|
rx={6}
|
||||||
|
/>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
import { BaseBoxShapeTool, TLEventHandlers } from 'tldraw'
|
||||||
|
|
||||||
|
export class ImageGenTool extends BaseBoxShapeTool {
|
||||||
|
static override id = 'ImageGen'
|
||||||
|
static override initial = 'idle'
|
||||||
|
override shapeType = 'ImageGen'
|
||||||
|
|
||||||
|
override onComplete: TLEventHandlers["onComplete"] = () => {
|
||||||
|
console.log('🎨 ImageGenTool: Shape creation completed')
|
||||||
|
this.editor.setCurrentTool('select')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -238,6 +238,7 @@ export function CustomContextMenu(props: TLUiContextMenuProps) {
|
||||||
<TldrawUiMenuItem {...tools.Transcription} disabled={hasSelection} />
|
<TldrawUiMenuItem {...tools.Transcription} disabled={hasSelection} />
|
||||||
<TldrawUiMenuItem {...tools.FathomMeetings} disabled={hasSelection} />
|
<TldrawUiMenuItem {...tools.FathomMeetings} disabled={hasSelection} />
|
||||||
<TldrawUiMenuItem {...tools.Holon} disabled={hasSelection} />
|
<TldrawUiMenuItem {...tools.Holon} disabled={hasSelection} />
|
||||||
|
<TldrawUiMenuItem {...tools.ImageGen} disabled={hasSelection} />
|
||||||
</TldrawUiMenuGroup>
|
</TldrawUiMenuGroup>
|
||||||
|
|
||||||
{/* Collections Group */}
|
{/* Collections Group */}
|
||||||
|
|
|
||||||
|
|
@ -29,7 +29,7 @@ export function CustomMainMenu() {
|
||||||
const validateAndNormalizeShapeType = (shape: any): string => {
|
const validateAndNormalizeShapeType = (shape: any): string => {
|
||||||
if (!shape || !shape.type) return 'text'
|
if (!shape || !shape.type) return 'text'
|
||||||
|
|
||||||
const validCustomShapes = ['ObsNote', 'VideoChat', 'Transcription', 'Prompt', 'ChatBox', 'Embed', 'Markdown', 'MycrozineTemplate', 'Slide', 'Holon', 'ObsidianBrowser', 'HolonBrowser', 'FathomMeetingsBrowser', 'LocationShare']
|
const validCustomShapes = ['ObsNote', 'VideoChat', 'Transcription', 'Prompt', 'ChatBox', 'Embed', 'Markdown', 'MycrozineTemplate', 'Slide', 'Holon', 'ObsidianBrowser', 'HolonBrowser', 'FathomMeetingsBrowser', 'LocationShare', 'ImageGen']
|
||||||
const validDefaultShapes = ['arrow', 'bookmark', 'draw', 'embed', 'frame', 'geo', 'group', 'highlight', 'image', 'line', 'note', 'text', 'video']
|
const validDefaultShapes = ['arrow', 'bookmark', 'draw', 'embed', 'frame', 'geo', 'group', 'highlight', 'image', 'line', 'note', 'text', 'video']
|
||||||
const allValidShapes = [...validCustomShapes, ...validDefaultShapes]
|
const allValidShapes = [...validCustomShapes, ...validDefaultShapes]
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ export const components: TLComponents = {
|
||||||
tools["Transcription"],
|
tools["Transcription"],
|
||||||
tools["Holon"],
|
tools["Holon"],
|
||||||
tools["FathomMeetings"],
|
tools["FathomMeetings"],
|
||||||
|
tools["ImageGen"],
|
||||||
].filter(tool => tool && tool.kbd)
|
].filter(tool => tool && tool.kbd)
|
||||||
|
|
||||||
// Get all custom actions with keyboard shortcuts
|
// Get all custom actions with keyboard shortcuts
|
||||||
|
|
|
||||||
|
|
@ -196,6 +196,15 @@ export const overrides: TLUiOverrides = {
|
||||||
// Shape creation is handled manually in FathomMeetingsTool.onPointerDown
|
// Shape creation is handled manually in FathomMeetingsTool.onPointerDown
|
||||||
onSelect: () => editor.setCurrentTool("fathom-meetings"),
|
onSelect: () => editor.setCurrentTool("fathom-meetings"),
|
||||||
},
|
},
|
||||||
|
ImageGen: {
|
||||||
|
id: "ImageGen",
|
||||||
|
icon: "image",
|
||||||
|
label: "Image Generation",
|
||||||
|
kbd: "alt+i",
|
||||||
|
readonlyOk: true,
|
||||||
|
type: "ImageGen",
|
||||||
|
onSelect: () => editor.setCurrentTool("ImageGen"),
|
||||||
|
},
|
||||||
hand: {
|
hand: {
|
||||||
...tools.hand,
|
...tools.hand,
|
||||||
onDoubleClick: (info: any) => {
|
onDoubleClick: (info: any) => {
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
import OpenAI from "openai";
|
import OpenAI from "openai";
|
||||||
import Anthropic from "@anthropic-ai/sdk";
|
import Anthropic from "@anthropic-ai/sdk";
|
||||||
import { makeRealSettings, AI_PERSONALITIES } from "@/lib/settings";
|
import { makeRealSettings, AI_PERSONALITIES } from "@/lib/settings";
|
||||||
|
import { getRunPodConfig } from "@/lib/clientConfig";
|
||||||
|
|
||||||
export async function llm(
|
export async function llm(
|
||||||
userPrompt: string,
|
userPrompt: string,
|
||||||
|
|
@ -59,7 +60,12 @@ export async function llm(
|
||||||
availableProviders.map(p => `${p.provider} (${p.model})`).join(', '));
|
availableProviders.map(p => `${p.provider} (${p.model})`).join(', '));
|
||||||
|
|
||||||
if (availableProviders.length === 0) {
|
if (availableProviders.length === 0) {
|
||||||
throw new Error("No valid API key found for any provider")
|
const runpodConfig = getRunPodConfig();
|
||||||
|
if (runpodConfig && runpodConfig.apiKey && runpodConfig.endpointId) {
|
||||||
|
// RunPod should have been added, but if not, try one more time
|
||||||
|
console.log('⚠️ No user API keys found, but RunPod is configured - this should not happen');
|
||||||
|
}
|
||||||
|
throw new Error("No valid API key found for any provider. Please configure API keys in settings or set up RunPod environment variables (VITE_RUNPOD_API_KEY and VITE_RUNPOD_ENDPOINT_ID).")
|
||||||
}
|
}
|
||||||
|
|
||||||
// Try each provider/key combination in order until one succeeds
|
// Try each provider/key combination in order until one succeeds
|
||||||
|
|
@ -76,13 +82,14 @@ export async function llm(
|
||||||
'claude-3-haiku-20240307',
|
'claude-3-haiku-20240307',
|
||||||
];
|
];
|
||||||
|
|
||||||
for (const { provider, apiKey, model } of availableProviders) {
|
for (const providerInfo of availableProviders) {
|
||||||
|
const { provider, apiKey, model, endpointId } = providerInfo as any;
|
||||||
try {
|
try {
|
||||||
console.log(`🔄 Attempting to use ${provider} API (${model})...`);
|
console.log(`🔄 Attempting to use ${provider} API (${model})...`);
|
||||||
attemptedProviders.push(`${provider} (${model})`);
|
attemptedProviders.push(`${provider} (${model})`);
|
||||||
|
|
||||||
// Add retry logic for temporary failures
|
// Add retry logic for temporary failures
|
||||||
await callProviderAPIWithRetry(provider, apiKey, model, userPrompt, onToken, settings);
|
await callProviderAPIWithRetry(provider, apiKey, model, userPrompt, onToken, settings, endpointId);
|
||||||
console.log(`✅ Successfully used ${provider} API (${model})`);
|
console.log(`✅ Successfully used ${provider} API (${model})`);
|
||||||
return; // Success, exit the function
|
return; // Success, exit the function
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
|
@ -100,7 +107,9 @@ export async function llm(
|
||||||
try {
|
try {
|
||||||
console.log(`🔄 Trying fallback model: ${fallbackModel}...`);
|
console.log(`🔄 Trying fallback model: ${fallbackModel}...`);
|
||||||
attemptedProviders.push(`${provider} (${fallbackModel})`);
|
attemptedProviders.push(`${provider} (${fallbackModel})`);
|
||||||
await callProviderAPIWithRetry(provider, apiKey, fallbackModel, userPrompt, onToken, settings);
|
const providerInfo = availableProviders.find(p => p.provider === provider);
|
||||||
|
const endpointId = (providerInfo as any)?.endpointId;
|
||||||
|
await callProviderAPIWithRetry(provider, apiKey, fallbackModel, userPrompt, onToken, settings, endpointId);
|
||||||
console.log(`✅ Successfully used ${provider} API with fallback model ${fallbackModel}`);
|
console.log(`✅ Successfully used ${provider} API with fallback model ${fallbackModel}`);
|
||||||
fallbackSucceeded = true;
|
fallbackSucceeded = true;
|
||||||
return; // Success, exit the function
|
return; // Success, exit the function
|
||||||
|
|
@ -142,13 +151,17 @@ function getAvailableProviders(availableKeys: Record<string, string>, settings:
|
||||||
const providers = [];
|
const providers = [];
|
||||||
|
|
||||||
// Helper to add a provider key if valid
|
// Helper to add a provider key if valid
|
||||||
const addProviderKey = (provider: string, apiKey: string, model?: string) => {
|
const addProviderKey = (provider: string, apiKey: string, model?: string, endpointId?: string) => {
|
||||||
if (isValidApiKey(provider, apiKey) && !isApiKeyInvalid(provider, apiKey)) {
|
if (isValidApiKey(provider, apiKey) && !isApiKeyInvalid(provider, apiKey)) {
|
||||||
providers.push({
|
const providerInfo: any = {
|
||||||
provider: provider,
|
provider: provider,
|
||||||
apiKey: apiKey,
|
apiKey: apiKey,
|
||||||
model: model || settings.models[provider] || getDefaultModel(provider)
|
model: model || settings.models[provider] || getDefaultModel(provider)
|
||||||
});
|
};
|
||||||
|
if (endpointId) {
|
||||||
|
providerInfo.endpointId = endpointId;
|
||||||
|
}
|
||||||
|
providers.push(providerInfo);
|
||||||
return true;
|
return true;
|
||||||
} else if (isApiKeyInvalid(provider, apiKey)) {
|
} else if (isApiKeyInvalid(provider, apiKey)) {
|
||||||
console.log(`⏭️ Skipping ${provider} API key (marked as invalid)`);
|
console.log(`⏭️ Skipping ${provider} API key (marked as invalid)`);
|
||||||
|
|
@ -156,6 +169,20 @@ function getAvailableProviders(availableKeys: Record<string, string>, settings:
|
||||||
return false;
|
return false;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// PRIORITY 1: Check for RunPod configuration from environment variables FIRST
|
||||||
|
// RunPod takes priority over user-configured keys
|
||||||
|
const runpodConfig = getRunPodConfig();
|
||||||
|
if (runpodConfig && runpodConfig.apiKey && runpodConfig.endpointId) {
|
||||||
|
console.log('🔑 Found RunPod configuration from environment variables - using as primary AI provider');
|
||||||
|
providers.push({
|
||||||
|
provider: 'runpod',
|
||||||
|
apiKey: runpodConfig.apiKey,
|
||||||
|
endpointId: runpodConfig.endpointId,
|
||||||
|
model: 'default' // RunPod doesn't use model selection in the same way
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// PRIORITY 2: Then add user-configured keys (they will be tried after RunPod)
|
||||||
// First, try the preferred provider - support multiple keys if stored as comma-separated
|
// First, try the preferred provider - support multiple keys if stored as comma-separated
|
||||||
if (settings.provider && availableKeys[settings.provider]) {
|
if (settings.provider && availableKeys[settings.provider]) {
|
||||||
const keyValue = availableKeys[settings.provider];
|
const keyValue = availableKeys[settings.provider];
|
||||||
|
|
@ -239,8 +266,10 @@ function getAvailableProviders(availableKeys: Record<string, string>, settings:
|
||||||
}
|
}
|
||||||
|
|
||||||
// Additional fallback: Check for user-specific API keys from profile dashboard
|
// Additional fallback: Check for user-specific API keys from profile dashboard
|
||||||
if (providers.length === 0) {
|
// These will be tried after RunPod (if RunPod was added)
|
||||||
providers.push(...getUserSpecificApiKeys());
|
const userSpecificKeys = getUserSpecificApiKeys();
|
||||||
|
if (userSpecificKeys.length > 0) {
|
||||||
|
providers.push(...userSpecificKeys);
|
||||||
}
|
}
|
||||||
|
|
||||||
return providers;
|
return providers;
|
||||||
|
|
@ -372,13 +401,14 @@ async function callProviderAPIWithRetry(
|
||||||
userPrompt: string,
|
userPrompt: string,
|
||||||
onToken: (partialResponse: string, done?: boolean) => void,
|
onToken: (partialResponse: string, done?: boolean) => void,
|
||||||
settings?: any,
|
settings?: any,
|
||||||
|
endpointId?: string,
|
||||||
maxRetries: number = 2
|
maxRetries: number = 2
|
||||||
) {
|
) {
|
||||||
let lastError: Error | null = null;
|
let lastError: Error | null = null;
|
||||||
|
|
||||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||||
try {
|
try {
|
||||||
await callProviderAPI(provider, apiKey, model, userPrompt, onToken, settings);
|
await callProviderAPI(provider, apiKey, model, userPrompt, onToken, settings, endpointId);
|
||||||
return; // Success
|
return; // Success
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
lastError = error as Error;
|
lastError = error as Error;
|
||||||
|
|
@ -471,12 +501,226 @@ async function callProviderAPI(
|
||||||
model: string,
|
model: string,
|
||||||
userPrompt: string,
|
userPrompt: string,
|
||||||
onToken: (partialResponse: string, done?: boolean) => void,
|
onToken: (partialResponse: string, done?: boolean) => void,
|
||||||
settings?: any
|
settings?: any,
|
||||||
|
endpointId?: string
|
||||||
) {
|
) {
|
||||||
let partial = "";
|
let partial = "";
|
||||||
const systemPrompt = settings ? getSystemPrompt(settings) : 'You are a helpful assistant.';
|
const systemPrompt = settings ? getSystemPrompt(settings) : 'You are a helpful assistant.';
|
||||||
|
|
||||||
if (provider === 'openai') {
|
if (provider === 'runpod') {
|
||||||
|
// RunPod API integration - uses environment variables for automatic setup
|
||||||
|
// Get endpointId from parameter or from config
|
||||||
|
let runpodEndpointId = endpointId;
|
||||||
|
if (!runpodEndpointId) {
|
||||||
|
const runpodConfig = getRunPodConfig();
|
||||||
|
if (runpodConfig) {
|
||||||
|
runpodEndpointId = runpodConfig.endpointId;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!runpodEndpointId) {
|
||||||
|
throw new Error('RunPod endpoint ID not configured');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try /runsync first for synchronous execution (returns output immediately)
|
||||||
|
// Fall back to /run + polling if /runsync is not available
|
||||||
|
const syncUrl = `https://api.runpod.ai/v2/${runpodEndpointId}/runsync`;
|
||||||
|
const asyncUrl = `https://api.runpod.ai/v2/${runpodEndpointId}/run`;
|
||||||
|
|
||||||
|
// vLLM endpoints typically expect OpenAI-compatible format with messages array
|
||||||
|
// But some endpoints might accept simple prompt format
|
||||||
|
// Try OpenAI-compatible format first, as it's more standard for vLLM
|
||||||
|
const messages = [];
|
||||||
|
if (systemPrompt) {
|
||||||
|
messages.push({ role: 'system', content: systemPrompt });
|
||||||
|
}
|
||||||
|
messages.push({ role: 'user', content: userPrompt });
|
||||||
|
|
||||||
|
// Combine system prompt and user prompt for simple prompt format (fallback)
|
||||||
|
const fullPrompt = systemPrompt ? `${systemPrompt}\n\nUser: ${userPrompt}` : userPrompt;
|
||||||
|
|
||||||
|
const requestBody = {
|
||||||
|
input: {
|
||||||
|
messages: messages,
|
||||||
|
stream: false // vLLM can handle streaming, but we'll process it synchronously for now
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
console.log('📤 RunPod API: Trying synchronous endpoint first:', syncUrl);
|
||||||
|
console.log('📤 RunPod API: Using OpenAI-compatible messages format');
|
||||||
|
|
||||||
|
try {
|
||||||
|
// First, try synchronous endpoint (/runsync) - this returns output immediately
|
||||||
|
try {
|
||||||
|
const syncResponse = await fetch(syncUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
},
|
||||||
|
body: JSON.stringify(requestBody)
|
||||||
|
});
|
||||||
|
|
||||||
|
if (syncResponse.ok) {
|
||||||
|
const syncData = await syncResponse.json();
|
||||||
|
console.log('📥 RunPod API: Synchronous response:', JSON.stringify(syncData, null, 2));
|
||||||
|
|
||||||
|
// Check if we got output directly
|
||||||
|
if (syncData.output) {
|
||||||
|
let responseText = '';
|
||||||
|
if (syncData.output.choices && Array.isArray(syncData.output.choices)) {
|
||||||
|
const choice = syncData.output.choices[0];
|
||||||
|
if (choice && choice.message && choice.message.content) {
|
||||||
|
responseText = choice.message.content;
|
||||||
|
}
|
||||||
|
} else if (typeof syncData.output === 'string') {
|
||||||
|
responseText = syncData.output;
|
||||||
|
} else if (syncData.output.text) {
|
||||||
|
responseText = syncData.output.text;
|
||||||
|
} else if (syncData.output.response) {
|
||||||
|
responseText = syncData.output.response;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (responseText) {
|
||||||
|
console.log('✅ RunPod API: Got output from synchronous endpoint, length:', responseText.length);
|
||||||
|
// Stream the response character by character to simulate streaming
|
||||||
|
for (let i = 0; i < responseText.length; i++) {
|
||||||
|
partial += responseText[i];
|
||||||
|
onToken(partial, false);
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 10));
|
||||||
|
}
|
||||||
|
onToken(partial, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// If sync endpoint returned a job ID, fall through to async polling
|
||||||
|
if (syncData.id && (syncData.status === 'IN_QUEUE' || syncData.status === 'IN_PROGRESS')) {
|
||||||
|
console.log('⏳ RunPod API: Sync endpoint returned job ID, polling:', syncData.id);
|
||||||
|
const result = await pollRunPodJob(syncData.id, apiKey, runpodEndpointId);
|
||||||
|
console.log('✅ RunPod API: Job completed, result length:', result.length);
|
||||||
|
partial = result;
|
||||||
|
onToken(partial, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (syncError) {
|
||||||
|
console.log('⚠️ RunPod API: Synchronous endpoint not available, trying async:', syncError);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fall back to async endpoint (/run) if sync didn't work
|
||||||
|
console.log('📤 RunPod API: Using async endpoint:', asyncUrl);
|
||||||
|
const response = await fetch(asyncUrl, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
},
|
||||||
|
body: JSON.stringify(requestBody)
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log('📥 RunPod API: Response status:', response.status, response.statusText);
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text();
|
||||||
|
console.error('❌ RunPod API: Error response:', errorText);
|
||||||
|
throw new Error(`RunPod API error: ${response.status} - ${errorText}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json();
|
||||||
|
console.log('📥 RunPod API: Response data:', JSON.stringify(data, null, 2));
|
||||||
|
|
||||||
|
// Handle async job pattern (RunPod often returns job IDs)
|
||||||
|
if (data.id && (data.status === 'IN_QUEUE' || data.status === 'IN_PROGRESS')) {
|
||||||
|
console.log('⏳ RunPod API: Job queued/in progress, polling job ID:', data.id);
|
||||||
|
const result = await pollRunPodJob(data.id, apiKey, runpodEndpointId);
|
||||||
|
console.log('✅ RunPod API: Job completed, result length:', result.length);
|
||||||
|
partial = result;
|
||||||
|
onToken(partial, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle OpenAI-compatible response format (vLLM endpoints)
|
||||||
|
if (data.output && data.output.choices && Array.isArray(data.output.choices)) {
|
||||||
|
console.log('📥 RunPod API: Detected OpenAI-compatible response format');
|
||||||
|
const choice = data.output.choices[0];
|
||||||
|
if (choice && choice.message && choice.message.content) {
|
||||||
|
const responseText = choice.message.content;
|
||||||
|
console.log('✅ RunPod API: Extracted content from OpenAI-compatible format, length:', responseText.length);
|
||||||
|
|
||||||
|
// Stream the response character by character to simulate streaming
|
||||||
|
for (let i = 0; i < responseText.length; i++) {
|
||||||
|
partial += responseText[i];
|
||||||
|
onToken(partial, false);
|
||||||
|
// Small delay to simulate streaming
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 10));
|
||||||
|
}
|
||||||
|
onToken(partial, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle direct response
|
||||||
|
if (data.output) {
|
||||||
|
console.log('📥 RunPod API: Processing output:', typeof data.output, Array.isArray(data.output) ? 'array' : 'object');
|
||||||
|
// Try to extract text from various possible response formats
|
||||||
|
let responseText = '';
|
||||||
|
if (typeof data.output === 'string') {
|
||||||
|
responseText = data.output;
|
||||||
|
console.log('✅ RunPod API: Extracted string output, length:', responseText.length);
|
||||||
|
} else if (data.output.text) {
|
||||||
|
responseText = data.output.text;
|
||||||
|
console.log('✅ RunPod API: Extracted text from output.text, length:', responseText.length);
|
||||||
|
} else if (data.output.response) {
|
||||||
|
responseText = data.output.response;
|
||||||
|
console.log('✅ RunPod API: Extracted response from output.response, length:', responseText.length);
|
||||||
|
} else if (data.output.content) {
|
||||||
|
responseText = data.output.content;
|
||||||
|
console.log('✅ RunPod API: Extracted content from output.content, length:', responseText.length);
|
||||||
|
} else if (Array.isArray(data.output.segments)) {
|
||||||
|
responseText = data.output.segments.map((seg: any) => seg.text || seg).join(' ');
|
||||||
|
console.log('✅ RunPod API: Extracted text from segments, length:', responseText.length);
|
||||||
|
} else {
|
||||||
|
// Fallback: stringify the output
|
||||||
|
console.warn('⚠️ RunPod API: Unknown output format, stringifying:', Object.keys(data.output));
|
||||||
|
responseText = JSON.stringify(data.output);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stream the response character by character to simulate streaming
|
||||||
|
for (let i = 0; i < responseText.length; i++) {
|
||||||
|
partial += responseText[i];
|
||||||
|
onToken(partial, false);
|
||||||
|
// Small delay to simulate streaming
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 10));
|
||||||
|
}
|
||||||
|
onToken(partial, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle error response
|
||||||
|
if (data.error) {
|
||||||
|
console.error('❌ RunPod API: Error in response:', data.error);
|
||||||
|
throw new Error(`RunPod API error: ${data.error}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check for status messages that might indicate endpoint is starting up
|
||||||
|
if (data.status) {
|
||||||
|
console.log('ℹ️ RunPod API: Response status:', data.status);
|
||||||
|
if (data.status === 'STARTING' || data.status === 'PENDING') {
|
||||||
|
console.log('⏳ RunPod API: Endpoint appears to be starting up, this may take a moment...');
|
||||||
|
// Wait a bit and retry
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 2000));
|
||||||
|
throw new Error('RunPod endpoint is starting up. Please wait a moment and try again.');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.error('❌ RunPod API: No valid response format detected. Full response:', JSON.stringify(data, null, 2));
|
||||||
|
throw new Error('No valid response from RunPod API');
|
||||||
|
} catch (error) {
|
||||||
|
console.error('❌ RunPod API error:', error);
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
} else if (provider === 'openai') {
|
||||||
const openai = new OpenAI({
|
const openai = new OpenAI({
|
||||||
apiKey,
|
apiKey,
|
||||||
dangerouslyAllowBrowser: true,
|
dangerouslyAllowBrowser: true,
|
||||||
|
|
@ -556,6 +800,185 @@ async function callProviderAPI(
|
||||||
onToken(partial, true);
|
onToken(partial, true);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Helper function to poll RunPod job status until completion
|
||||||
|
async function pollRunPodJob(
|
||||||
|
jobId: string,
|
||||||
|
apiKey: string,
|
||||||
|
endpointId: string,
|
||||||
|
maxAttempts: number = 60,
|
||||||
|
pollInterval: number = 1000
|
||||||
|
): Promise<string> {
|
||||||
|
const statusUrl = `https://api.runpod.ai/v2/${endpointId}/status/${jobId}`;
|
||||||
|
console.log('🔄 RunPod API: Starting to poll job:', jobId);
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
||||||
|
try {
|
||||||
|
const response = await fetch(statusUrl, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorText = await response.text();
|
||||||
|
console.error(`❌ RunPod API: Poll error (attempt ${attempt + 1}/${maxAttempts}):`, response.status, errorText);
|
||||||
|
throw new Error(`Failed to check job status: ${response.status} - ${errorText}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json();
|
||||||
|
console.log(`🔄 RunPod API: Poll attempt ${attempt + 1}/${maxAttempts}, status:`, data.status);
|
||||||
|
console.log(`📥 RunPod API: Full poll response:`, JSON.stringify(data, null, 2));
|
||||||
|
|
||||||
|
if (data.status === 'COMPLETED') {
|
||||||
|
console.log('✅ RunPod API: Job completed, processing output...');
|
||||||
|
console.log('📥 RunPod API: Output structure:', typeof data.output, data.output ? Object.keys(data.output) : 'null');
|
||||||
|
console.log('📥 RunPod API: Full data object keys:', Object.keys(data));
|
||||||
|
|
||||||
|
// If no output after a couple of retries, try the stream endpoint as fallback
|
||||||
|
if (!data.output) {
|
||||||
|
if (attempt < 3) {
|
||||||
|
// Only retry 2-3 times, then try stream endpoint
|
||||||
|
console.log(`⏳ RunPod API: COMPLETED but no output yet, waiting briefly (attempt ${attempt + 1}/3)...`);
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 500));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// After a few retries, try the stream endpoint as fallback
|
||||||
|
console.log('⚠️ RunPod API: Status endpoint not returning output, trying stream endpoint...');
|
||||||
|
try {
|
||||||
|
const streamUrl = `https://api.runpod.ai/v2/${endpointId}/stream/${jobId}`;
|
||||||
|
const streamResponse = await fetch(streamUrl, {
|
||||||
|
method: 'GET',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
if (streamResponse.ok) {
|
||||||
|
const streamData = await streamResponse.json();
|
||||||
|
console.log('📥 RunPod API: Stream endpoint response:', JSON.stringify(streamData, null, 2));
|
||||||
|
|
||||||
|
if (streamData.output) {
|
||||||
|
// Use stream endpoint output
|
||||||
|
data.output = streamData.output;
|
||||||
|
console.log('✅ RunPod API: Found output via stream endpoint');
|
||||||
|
} else if (streamData.choices && Array.isArray(streamData.choices)) {
|
||||||
|
// Handle OpenAI-compatible format from stream endpoint
|
||||||
|
data.output = { choices: streamData.choices };
|
||||||
|
console.log('✅ RunPod API: Found choices via stream endpoint');
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log(`⚠️ RunPod API: Stream endpoint returned ${streamResponse.status}`);
|
||||||
|
}
|
||||||
|
} catch (streamError) {
|
||||||
|
console.log('⚠️ RunPod API: Stream endpoint not available or failed:', streamError);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract text from various possible response formats
|
||||||
|
let result = '';
|
||||||
|
if (typeof data.output === 'string') {
|
||||||
|
result = data.output;
|
||||||
|
console.log('✅ RunPod API: Extracted string output from job, length:', result.length);
|
||||||
|
} else if (data.output?.text) {
|
||||||
|
result = data.output.text;
|
||||||
|
console.log('✅ RunPod API: Extracted text from output.text, length:', result.length);
|
||||||
|
} else if (data.output?.response) {
|
||||||
|
result = data.output.response;
|
||||||
|
console.log('✅ RunPod API: Extracted response from output.response, length:', result.length);
|
||||||
|
} else if (data.output?.content) {
|
||||||
|
result = data.output.content;
|
||||||
|
console.log('✅ RunPod API: Extracted content from output.content, length:', result.length);
|
||||||
|
} else if (data.output?.choices && Array.isArray(data.output.choices)) {
|
||||||
|
// Handle OpenAI-compatible response format (vLLM endpoints)
|
||||||
|
const choice = data.output.choices[0];
|
||||||
|
if (choice && choice.message && choice.message.content) {
|
||||||
|
result = choice.message.content;
|
||||||
|
console.log('✅ RunPod API: Extracted content from OpenAI-compatible format, length:', result.length);
|
||||||
|
}
|
||||||
|
} else if (data.output?.segments && Array.isArray(data.output.segments)) {
|
||||||
|
result = data.output.segments.map((seg: any) => seg.text || seg).join(' ');
|
||||||
|
console.log('✅ RunPod API: Extracted text from segments, length:', result.length);
|
||||||
|
} else if (Array.isArray(data.output)) {
|
||||||
|
// Handle array responses (some vLLM endpoints return arrays)
|
||||||
|
result = data.output.map((item: any) => {
|
||||||
|
if (typeof item === 'string') return item;
|
||||||
|
if (item.text) return item.text;
|
||||||
|
if (item.response) return item.response;
|
||||||
|
return JSON.stringify(item);
|
||||||
|
}).join('\n');
|
||||||
|
console.log('✅ RunPod API: Extracted text from array output, length:', result.length);
|
||||||
|
} else if (!data.output) {
|
||||||
|
// No output field - check alternative structures or return empty
|
||||||
|
console.warn('⚠️ RunPod API: No output field found, checking alternative structures...');
|
||||||
|
console.log('📥 RunPod API: Full data structure:', JSON.stringify(data, null, 2));
|
||||||
|
|
||||||
|
// Try checking if output is directly in data (not data.output)
|
||||||
|
if (typeof data === 'string') {
|
||||||
|
result = data;
|
||||||
|
console.log('✅ RunPod API: Data itself is a string, length:', result.length);
|
||||||
|
} else if (data.text) {
|
||||||
|
result = data.text;
|
||||||
|
console.log('✅ RunPod API: Found text at top level, length:', result.length);
|
||||||
|
} else if (data.response) {
|
||||||
|
result = data.response;
|
||||||
|
console.log('✅ RunPod API: Found response at top level, length:', result.length);
|
||||||
|
} else if (data.content) {
|
||||||
|
result = data.content;
|
||||||
|
console.log('✅ RunPod API: Found content at top level, length:', result.length);
|
||||||
|
} else {
|
||||||
|
// Stream endpoint already tried above (around line 848), just log that we couldn't find output
|
||||||
|
if (attempt >= 3) {
|
||||||
|
console.warn('⚠️ RunPod API: Could not find output in status or stream endpoint after multiple attempts');
|
||||||
|
}
|
||||||
|
|
||||||
|
// If still no result, return empty string instead of throwing error
|
||||||
|
// This allows the UI to render something instead of failing
|
||||||
|
if (!result) {
|
||||||
|
console.warn('⚠️ RunPod API: No output found in response. Returning empty result.');
|
||||||
|
console.log('📥 RunPod API: Available fields:', Object.keys(data));
|
||||||
|
result = ''; // Return empty string so UI can render
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return result even if empty - don't loop forever
|
||||||
|
if (result !== undefined) {
|
||||||
|
// Return empty string if no result found - allows UI to render
|
||||||
|
console.log('✅ RunPod API: Returning result (may be empty):', result ? `length ${result.length}` : 'empty');
|
||||||
|
return result || '';
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we get here, no output was found - return empty string instead of looping
|
||||||
|
console.warn('⚠️ RunPod API: No output found after checking all formats. Returning empty result.');
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (data.status === 'FAILED') {
|
||||||
|
console.error('❌ RunPod API: Job failed:', data.error || 'Unknown error');
|
||||||
|
throw new Error(`Job failed: ${data.error || 'Unknown error'}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check for starting/pending status
|
||||||
|
if (data.status === 'STARTING' || data.status === 'PENDING') {
|
||||||
|
console.log(`⏳ RunPod API: Endpoint still starting (attempt ${attempt + 1}/${maxAttempts})...`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Job still in progress, wait and retry
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval));
|
||||||
|
} catch (error) {
|
||||||
|
if (attempt === maxAttempts - 1) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
// Wait before retrying
|
||||||
|
await new Promise(resolve => setTimeout(resolve, pollInterval));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error('Job polling timeout - job did not complete in time');
|
||||||
|
}
|
||||||
|
|
||||||
// Auto-migration function that runs automatically
|
// Auto-migration function that runs automatically
|
||||||
async function autoMigrateAPIKeys() {
|
async function autoMigrateAPIKeys() {
|
||||||
try {
|
try {
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue