XFEstudio/gpt4free
feat: add EdgeTTS audio provider and global image→media refactor
- **Docs** - `docs/file.md`: update upload instructions to use inline `bucket` content parts instead of `tool_calls/bucket_tool`. - `docs/media.md`: add asynchronous audio transcription example, detailed explanation, and notes. - **New audio provider** - Add `g4f/Provider/audio/EdgeTTS.py` implementing Edge Text‑to‑Speech (`EdgeTTS`). - Create `g4f/Provider/audio/__init__.py` for provider export. - Register provider in `g4f/Provider/__init__.py`. - **Refactor image → media** - Introduce `generated_media/` directory and `get_media_dir()` helper in `g4f/image/copy_images.py`; add `ensure_media_dir()`; keep back‑compat with legacy `generated_images/`. - Replace `images_dir` references with `get_media_dir()` across: - `g4f/api/__init__.py` - `g4f/client/stubs.py` - `g4f/gui/server/api.py` - `g4f/gui/server/backend_api.py` - `g4f/image/copy_images.py` - Rename CLI/API config field/flag from `image_provider` to `media_provider` (`g4f/cli.py`, `g4f/api/__init__.py`, `g4f/client/__init__.py`). - Extend `g4f/image/__init__.py` - add `MEDIA_TYPE_MAP`, `get_extension()` - revise `is_allowed_extension()`, `to_input_audio()` to support wider media types. - **Provider adjustments** - `g4f/Provider/ARTA.py`: swap `raise_error()` parameter order. - `g4f/Provider/Cloudflare.py`: drop unused `MissingRequirementsError` import; move `get_args_from_nodriver()` inside try; handle `FileNotFoundError`. - **Core enhancements** - `g4f/providers/any_provider.py`: use `default_model` instead of literal `"default"`; broaden model/provider matching; update model list cleanup. - `g4f/models.py`: safeguard provider count logic when model name is falsy. - `g4f/providers/base_provider.py`: catch `json.JSONDecodeError` when reading auth cache, delete corrupted file. - `g4f/providers/response.py`: allow `AudioResponse` to accept extra kwargs. - **Misc** - Remove obsolete `g4f/image.py`. - `g4f/Provider/Cloudflare.py`, `g4f/client/types.py`: minor whitespace and import tidy‑ups.
e83282fc
代码差异
@@ -180,23 +180,17 @@ fileInput.addEventListener('change', () => {
**Integrating with `ChatCompletion`:**
To incorporate file uploads into your client applications, include the `tool_calls` parameter in your chat completion requests, using the `bucket_tool` function. The `bucket_id` is passed as a JSON object within your prompt.
To incorporate file uploads into your client applications, include the `bucket` in your chat completion requests, using inline content parts.
```json
{
"messages": [
{
"role": "user",
"content": "Answer this question using the files in the specified bucket: ...your question...\n{\"bucket_id\": \"your_actual_bucket_id\"}"
}
],
"tool_calls": [
{
"function": {
"name": "bucket_tool"
},
"type": "function"
"content": [
{"type": "text", "text": "Answer this question using the files in the specified bucket: ...your question..."},
{"bucket_id": "your_actual_bucket_id"}
]
}
]
}
@@ -30,6 +30,8 @@ asyncio.run(main())
#### **Transcribe an Audio File:**
Some providers in G4F support audio inputs in chat completions, allowing you to transcribe audio files by instructing the model accordingly. This example demonstrates how to use the `AsyncClient` to transcribe an audio file asynchronously:
```python
import asyncio
from g4f.client import AsyncClient
@@ -41,15 +43,32 @@ async def main():
with open("audio.wav", "rb") as audio_file:
response = await client.chat.completions.create(
messages="Transcribe this audio",
provider=g4f.Provider.Microsoft_Phi_4,
media=[[audio_file, "audio.wav"]],
modalities=["text"],
)
print(response.choices[0].message.content)
asyncio.run(main())
print(response.choices[0].message.content)
if __name__ == "__main__":
asyncio.run(main())
```
#### Explanation
- **Client Initialization**: An `AsyncClient` instance is created with a provider that supports audio inputs, such as `PollinationsAI` or `Microsoft_Phi_4`.
- **File Handling**: The audio file (`audio.wav`) is opened in binary read mode (`"rb"`) using a context manager (`with` statement) to ensure proper file closure after use.
- **API Call**: The `chat.completions.create` method is called with:
- `messages`: Containing a user message instructing the model to transcribe the audio.
- `media`: A list of lists, where each inner list contains the file object and its name (`[[audio_file, "audio.wav"]]`).
- `modalities=["text"]`: Specifies that the output should be text (the transcription).
- **Response**: The transcription is extracted from `response.choices[0].message.content` and printed.
#### Notes
- **Provider Support**: Ensure the chosen provider (e.g., `PollinationsAI` or `Microsoft_Phi_4`) supports audio inputs in chat completions. Not all providers may offer this functionality.
- **File Path**: Replace `"audio.wav"` with the path to your own audio file. The file format (e.g., WAV) should be compatible with the provider.
- **Model Selection**: If `g4f.models.default` does not support audio transcription, you may need to specify a model that does (consult the provider's documentation for supported models).
This example complements the guide by showcasing how to handle audio inputs asynchronously, expanding on the multimodal capabilities of the G4F AsyncClient API.
---
### 2. **Image Generation**
@@ -203,7 +203,7 @@ class ARTA(AsyncGeneratorProvider, ProviderModelMixin):
else:
raise ResponseError(f"Image generation failed with status: {status}")
async def raise_error(response: ClientResponse, message: str):
async def raise_error(message: str, response: ClientResponse):
if response.ok:
return
error_text = await response.text()
@@ -20,7 +20,7 @@ from ..cookies import get_cookies_dir
from .helper import format_image_prompt, render_messages
from ..providers.response import JsonConversation, ImageResponse
from ..tools.media import merge_media
from ..errors import RateLimitError
from ..errors import RateLimitError, NoValidHarFileError
from .. import debug
class Conversation(JsonConversation):
@@ -470,6 +470,8 @@ class Blackbox(AsyncGeneratorProvider, ProviderModelMixin):
except Exception as e:
debug.log(f"Blackbox: Error reading HAR file {file}: {e}")
return None
except NoValidHarFileError:
pass
except Exception as e:
debug.log(f"Blackbox: Error searching HAR files: {e}")
return None
@@ -8,7 +8,7 @@ from .base_provider import AsyncGeneratorProvider, ProviderModelMixin, AuthFileM
from ..requests import Session, StreamSession, get_args_from_nodriver, raise_for_status, merge_cookies
from ..requests import DEFAULT_HEADERS, has_nodriver, has_curl_cffi
from ..providers.response import FinishReason, Usage
from ..errors import ResponseStatusError, ModelNotFoundError, MissingRequirementsError
from ..errors import ResponseStatusError, ModelNotFoundError
from .. import debug
from .helper import render_messages
@@ -72,11 +72,11 @@ class Cloudflare(AsyncGeneratorProvider, ProviderModelMixin, AuthFileMixin):
except ResponseStatusError as f:
if has_nodriver:
get_running_loop(check_nested=True)
args = get_args_from_nodriver(cls.url)
try:
args = get_args_from_nodriver(cls.url)
cls._args = asyncio.run(args)
read_models()
except RuntimeError as e:
except (RuntimeError, FileNotFoundError) as e:
cls.models = cls.fallback_models
debug.log(f"Nodriver is not available: {type(e).__name__}: {e}")
else:
@@ -28,6 +28,10 @@ try:
from .mini_max import HailuoAI, MiniMax
except ImportError as e:
debug.error("MiniMax providers not loaded:", e)
try:
from .audio import EdgeTTS
except ImportError as e:
debug.error("Audio providers not loaded:", e)
try:
from .AllenAI import AllenAI
@@ -0,0 +1,75 @@
from __future__ import annotations
import os
import random
import asyncio
try:
import edge_tts
from edge_tts import VoicesManager
has_edge_tts = True
except ImportError:
has_edge_tts = False
from ...typing import AsyncResult, Messages
from ...providers.response import AudioResponse
from ...image.copy_images import get_filename, get_media_dir, ensure_media_dir
from ..base_provider import AsyncGeneratorProvider, ProviderModelMixin
from ..helper import format_image_prompt
class EdgeTTS(AsyncGeneratorProvider, ProviderModelMixin):
label = "Edge TTS"
working = has_edge_tts
default_model = "edge-tts"
default_locale = "en-US"
@classmethod
def get_models(cls) -> list[str]:
if not cls.models:
voices = asyncio.run(VoicesManager.create())
cls.default_model = voices.find(Locale=cls.default_locale)[0]["Name"]
cls.models = [voice["Name"] for voice in voices.voices]
return cls.models
@classmethod
async def create_async_generator(
cls,
model: str,
messages: Messages,
proxy: str = None,
prompt: str = None,
language: str = None,
locale: str = None,
audio: dict = {"voice": None, "format": "mp3"},
extra_parameters: list[str] = ["rate", "volume", "pitch"],
**kwargs
) -> AsyncResult:
prompt = format_image_prompt(messages, prompt)
if not prompt:
raise ValueError("Prompt is empty.")
voice = audio.get("voice", model)
if not voice:
voices = await VoicesManager.create()
if locale is None:
if language is None:
voices = voices.find(Locale=cls.default_locale)
elif "-" in language:
voices = voices.find(Locale=language)
else:
voices = voices.find(Language=language)
else:
voices = voices.find(Locale=locale)
if not voices:
raise ValueError(f"No voices found for language '{language}' and locale '{locale}'.")
voice = random.choice(voices)["Name"]
format = audio.get("format", "mp3")
filename = get_filename([cls.default_model], prompt, f".{format}", prompt)
target_path = os.path.join(get_media_dir(), filename)
ensure_media_dir()
extra_parameters = {param: kwargs[param] for param in extra_parameters if param in kwargs}
communicate = edge_tts.Communicate(prompt, voice=voice, proxy=proxy, **extra_parameters)
await communicate.save(target_path)
yield AudioResponse(f"/media/{filename}", voice=voice, prompt=prompt)
@@ -0,0 +1 @@
from .EdgeTTS import EdgeTTS
@@ -40,7 +40,7 @@ from g4f.client import AsyncClient, ChatCompletion, ImagesResponse, convert_to_p
from g4f.providers.response import BaseConversation, JsonConversation
from g4f.client.helper import filter_none
from g4f.image import is_data_an_media, EXTENSIONS_MAP
from g4f.image.copy_images import images_dir, copy_media, get_source_url
from g4f.image.copy_images import get_media_dir, copy_media, get_source_url
from g4f.errors import ProviderNotFoundError, ModelNotFoundError, MissingAuthError, NoValidHarFileError
from g4f.cookies import read_cookie_files, get_cookies_dir
from g4f.providers.types import ProviderType
@@ -130,7 +130,7 @@ class AppConfig:
ignore_cookie_files: bool = False
model: str = None
provider: str = None
image_provider: str = None
media_provider: str = None
proxy: str = None
gui: bool = False
demo: bool = False
@@ -419,12 +419,13 @@ class Api:
):
if config.provider is None:
config.provider = provider
if config.provider is None:
config.provider = AppConfig.media_provider
if credentials is not None and credentials.credentials != "secret":
config.api_key = credentials.credentials
try:
response = await self.client.images.generate(
**config.dict(exclude_none=True),
provider=AppConfig.image_provider if config.provider is None else config.provider
)
for image in response.data:
if hasattr(image, "url") and image.url.startswith("/"):
@@ -562,9 +563,9 @@ class Api:
HTTP_404_NOT_FOUND: {}
})
async def get_media(filename, request: Request):
target = os.path.join(images_dir, os.path.basename(filename))
target = os.path.join(get_media_dir(), os.path.basename(filename))
if not os.path.isfile(target):
other_name = os.path.join(images_dir, os.path.basename(quote_plus(filename)))
other_name = os.path.join(get_media_dir(), os.path.basename(quote_plus(filename)))
if os.path.isfile(other_name):
target = other_name
ext = os.path.splitext(filename)[1][1:]
@@ -627,7 +628,7 @@ class Api:
def format_exception(e: Union[Exception, str], config: Union[ChatCompletionsConfig, ImageGenerationConfig] = None, image: bool = False) -> str:
last_provider = {} if not image else g4f.get_last_provider(True)
provider = (AppConfig.image_provider if image else AppConfig.provider)
provider = (AppConfig.media_provider if image else AppConfig.provider)
model = AppConfig.model
if config is not None:
if config.provider is not None:
@@ -16,7 +16,7 @@ def get_api_parser():
api_parser.add_argument("--model", default=None, help="Default model for chat completion. (incompatible with --reload and --workers)")
api_parser.add_argument("--provider", choices=[provider.__name__ for provider in Provider.__providers__ if provider.working],
default=None, help="Default provider for chat completion. (incompatible with --reload and --workers)")
api_parser.add_argument("--image-provider", choices=[provider.__name__ for provider in Provider.__providers__ if provider.working and hasattr(provider, "image_models")],
api_parser.add_argument("--media-provider", choices=[provider.__name__ for provider in Provider.__providers__ if provider.working and bool(getattr(provider, "image_models", False))],
default=None, help="Default provider for image generation. (incompatible with --reload and --workers)"),
api_parser.add_argument("--proxy", default=None, help="Default used proxy. (incompatible with --reload and --workers)")
api_parser.add_argument("--workers", type=int, default=None, help="Number of workers.")
@@ -59,7 +59,7 @@ def run_api_args(args):
ignored_providers=args.ignored_providers,
g4f_api_key=args.g4f_api_key,
provider=args.provider,
image_provider=args.image_provider,
media_provider=args.media_provider,
proxy=args.proxy,
model=args.model,
gui=args.gui,
@@ -1,5 +1,6 @@
from __future__ import annotations
import os
import time
import random
import string
@@ -8,7 +9,7 @@ import aiohttp
import base64
from typing import Union, AsyncIterator, Iterator, Awaitable, Optional
from ..image.copy_images import copy_media
from ..image.copy_images import copy_media, get_media_dir
from ..typing import Messages, ImageType
from ..providers.types import ProviderType, BaseRetryProvider
from ..providers.response import *
@@ -16,11 +17,11 @@ from ..errors import NoMediaResponseError
from ..providers.retry_provider import IterListProvider
from ..providers.asyncio import to_sync_generator
from ..providers.any_provider import AnyProvider
from ..Provider.needs_auth import BingCreateImages, OpenaiAccount
from ..Provider import OpenaiAccount, PollinationsImage
from ..tools.run_tools import async_iter_run_tools, iter_run_tools
from .stubs import ChatCompletion, ChatCompletionChunk, Image, ImagesResponse, UsageModel, ToolCallModel
from .models import ClientModels
from .types import IterResponse, ImageProvider, Client as BaseClient
from .types import IterResponse, Client as BaseClient
from .service import convert_to_provider
from .helper import find_stop, filter_json, filter_none, safe_aclose
from .. import debug
@@ -261,15 +262,15 @@ class Client(BaseClient):
def __init__(
self,
provider: Optional[ProviderType] = None,
image_provider: Optional[ImageProvider] = None,
media_provider: Optional[ProviderType] = None,
**kwargs
) -> None:
super().__init__(**kwargs)
self.chat: Chat = Chat(self, provider)
if image_provider is None:
image_provider = provider
self.models: ClientModels = ClientModels(self, provider, image_provider)
self.images: Images = Images(self, image_provider)
if media_provider is None:
media_provider = kwargs.get("image_provider", provider)
self.models: ClientModels = ClientModels(self, provider, media_provider)
self.images: Images = Images(self, media_provider)
self.media: Images = self.images
class Completions:
@@ -364,7 +365,7 @@ class Images:
"""
return asyncio.run(self.async_generate(prompt, model, provider, response_format, proxy, **kwargs))
async def get_provider_handler(self, model: Optional[str], provider: Optional[ImageProvider], default: ImageProvider) -> ImageProvider:
async def get_provider_handler(self, model: Optional[str], provider: Optional[ProviderType], default: ProviderType) -> ProviderType:
if provider is None:
provider_handler = self.provider
if provider_handler is None:
@@ -387,7 +388,7 @@ class Images:
api_key: Optional[str] = None,
**kwargs
) -> ImagesResponse:
provider_handler = await self.get_provider_handler(model, provider, BingCreateImages)
provider_handler = await self.get_provider_handler(model, provider, PollinationsImage)
provider_name = provider_handler.__name__ if hasattr(provider_handler, "__name__") else type(provider_handler).__name__
if proxy is None:
proxy = self.client.proxy
@@ -407,20 +408,17 @@ class Images:
debug.error(f"{provider.__name__} {type(e).__name__}: {e}")
else:
response = await self._generate_image_response(provider_handler, provider_name, model, prompt, proxy=proxy, api_key=api_key, **kwargs)
if isinstance(response, MediaResponse):
return await self._process_image_response(
response,
model,
provider_name,
response_format,
proxy
)
if response is None:
if error is not None:
raise error
raise NoMediaResponseError(f"No image response from {provider_name}")
raise NoMediaResponseError(f"Unexpected response type: {type(response)}")
raise NoMediaResponseError(f"No media response from {provider_name}")
return await self._process_image_response(
response,
model,
provider_name,
response_format,
proxy
)
async def _generate_image_response(
self,
@@ -441,7 +439,7 @@ class Images:
prompt=prompt,
**kwargs
):
if isinstance(item, MediaResponse):
if isinstance(item, (MediaResponse, AudioResponse)):
items.append(item)
elif hasattr(provider_handler, "create_completion"):
for item in provider_handler.create_completion(
@@ -451,13 +449,15 @@ class Images:
prompt=prompt,
**kwargs
):
if isinstance(item, MediaResponse):
if isinstance(item, (MediaResponse, AudioResponse)):
items.append(item)
else:
raise ValueError(f"Provider {provider_name} does not support image generation")
urls = []
for item in items:
if isinstance(item.urls, str):
if isinstance(item, AudioResponse):
urls.append(item.to_uri())
elif isinstance(item.urls, str):
urls.append(item.urls)
elif isinstance(item.urls, list):
urls.extend(item.urls)
@@ -508,14 +508,11 @@ class Images:
debug.error(f"{provider.__name__} {type(e).__name__}: {e}")
else:
response = await self._generate_image_response(provider_handler, provider_name, model, prompt, **kwargs)
if isinstance(response, MediaResponse):
return await self._process_image_response(response, model, provider_name, response_format, proxy)
if response is None:
if error is not None:
raise error
raise NoMediaResponseError(f"No image response from {provider_name}")
raise NoMediaResponseError(f"Unexpected response type: {type(response)}")
raise NoMediaResponseError(f"No media response from {provider_name}")
return await self._process_image_response(response, model, provider_name, response_format, proxy)
async def _process_image_response(
self,
@@ -531,12 +528,16 @@ class Images:
elif response_format == "b64_json":
# Convert URLs directly to base64 without saving
async def get_b64_from_url(url: str) -> Image:
if url.startswith("/media/"):
with open(os.path.join(get_media_dir(), os.path.basename(url)), "wb") as f:
b64_data = base64.b64encode(f.read()).decode()
return Image.model_construct(b64_json=b64_data, revised_prompt=response.alt)
async with aiohttp.ClientSession(cookies=response.get("cookies")) as session:
async with session.get(url, proxy=proxy) as resp:
if resp.status == 200:
image_data = await resp.read()
b64_data = base64.b64encode(image_data).decode()
b64_data = base64.b64encode(await resp.read()).decode()
return Image.model_construct(b64_json=b64_data, revised_prompt=response.alt)
return Image.model_construct(url=url, revised_prompt=response.alt)
images = await asyncio.gather(*[get_b64_from_url(image) for image in response.get_list()])
else:
# Save locally for None (default) case
@@ -554,15 +555,15 @@ class AsyncClient(BaseClient):
def __init__(
self,
provider: Optional[ProviderType] = None,
image_provider: Optional[ImageProvider] = None,
media_provider: Optional[ProviderType] = None,
**kwargs
) -> None:
super().__init__(**kwargs)
self.chat: AsyncChat = AsyncChat(self, provider)
if image_provider is None:
image_provider = provider
self.models: ClientModels = ClientModels(self, provider, image_provider)
self.images: AsyncImages = AsyncImages(self, image_provider)
if media_provider is None:
media_provider = kwargs.get("image_provider", provider)
self.models: ClientModels = ClientModels(self, provider, media_provider)
self.images: AsyncImages = AsyncImages(self, media_provider)
self.media: AsyncImages = self.images
class AsyncChat:
@@ -5,7 +5,7 @@ from typing import Optional, List
from time import time
from ..image import extract_data_uri
from ..image.copy_images import images_dir
from ..image.copy_images import get_media_dir
from ..client.helper import filter_markdown
from .helper import filter_none
@@ -123,7 +123,7 @@ class ChatCompletionMessage(BaseModel):
def save(self, filepath: str, allowd_types = None):
if hasattr(self.content, "data"):
os.rename(self.content.data.replace("/media", images_dir), filepath)
os.rename(self.content.data.replace("/media", get_media_dir()), filepath)
return
if self.content.startswith("data:"):
with open(filepath, "wb") as f: