feat: 实现素材类型区分展示功能
- 图片素材:直接展示图片内容,支持多种格式(JPG/PNG/GIF等) - 视频素材:展示缩略图,使用FFmpeg生成首帧 - 音频素材:提供播放控件,支持点击播放/暂停 后端改进: - 扩展get_material_thumbnail_base64命令支持图片类型 - 新增get_audio_file_base64命令处理音频文件访问 - 自动检测文件MIME类型,安全的文件访问机制 前端改进: - MaterialThumbnail组件重构支持三种素材类型 - 集成HTML5 Audio API实现音频播放 - 改进懒加载和缓存机制,优化性能 遵循promptx/tauri-desktop-app-expert开发规范
This commit is contained in:
parent
5102923feb
commit
eb47a2b8fb
|
|
@ -0,0 +1,118 @@
|
|||
# 素材类型展示功能开发
|
||||
|
||||
## 功能概述
|
||||
|
||||
根据promptx/tauri-desktop-app-expert规定的开发规范,实现了素材管理的类型区分展示功能:
|
||||
|
||||
### 1. 图片素材展示
|
||||
- **功能**:直接展示图片内容
|
||||
- **实现**:通过后端API `get_material_thumbnail_base64` 获取图片的base64数据
|
||||
- **支持格式**:JPG, JPEG, PNG, GIF, WEBP, BMP, TIFF, SVG
|
||||
- **特点**:
|
||||
- 懒加载机制,只有当图片可见时才加载
|
||||
- 缓存机制,避免重复请求
|
||||
- 自动检测MIME类型
|
||||
- 保持原始宽高比
|
||||
|
||||
### 2. 视频素材展示
|
||||
- **功能**:展示视频缩略图
|
||||
- **实现**:使用FFmpeg生成视频首帧缩略图
|
||||
- **特点**:
|
||||
- 自动生成160px宽度的缩略图
|
||||
- 保持视频原始宽高比
|
||||
- 缓存生成的缩略图文件
|
||||
- 支持重试机制
|
||||
|
||||
### 3. 音频素材展示
|
||||
- **功能**:提供音频播放控件
|
||||
- **实现**:通过后端API `get_audio_file_base64` 获取音频数据并播放
|
||||
- **支持格式**:MP3, WAV, FLAC, AAC, OGG, WMA, M4A
|
||||
- **特点**:
|
||||
- 点击播放/暂停控制
|
||||
- 播放状态指示
|
||||
- 音频数据通过base64传输
|
||||
- 播放结束自动重置状态
|
||||
|
||||
## 技术实现
|
||||
|
||||
### 后端改进
|
||||
|
||||
1. **扩展 `get_material_thumbnail_base64` 命令**
|
||||
- 添加对图片类型的支持
|
||||
- 根据素材类型分别处理
|
||||
- 自动检测文件MIME类型
|
||||
|
||||
2. **新增 `get_audio_file_base64` 命令**
|
||||
- 专门处理音频文件访问
|
||||
- 支持多种音频格式
|
||||
- 安全的文件访问机制
|
||||
|
||||
### 前端改进
|
||||
|
||||
1. **MaterialThumbnail组件增强**
|
||||
- 支持三种素材类型的不同展示方式
|
||||
- 音频播放控件集成
|
||||
- 改进的错误处理和加载状态
|
||||
|
||||
2. **音频播放功能**
|
||||
- 使用HTML5 Audio API
|
||||
- 播放状态管理
|
||||
- 用户友好的播放界面
|
||||
|
||||
## 文件修改列表
|
||||
|
||||
### 后端文件
|
||||
- `apps/desktop/src-tauri/src/presentation/commands/material_commands.rs`
|
||||
- 扩展 `get_material_thumbnail_base64` 支持图片
|
||||
- 新增 `get_audio_file_base64` 命令
|
||||
- `apps/desktop/src-tauri/src/lib.rs`
|
||||
- 注册新的音频API命令
|
||||
|
||||
### 前端文件
|
||||
- `apps/desktop/src/components/MaterialThumbnail.tsx`
|
||||
- 重构组件支持三种素材类型
|
||||
- 添加音频播放功能
|
||||
- 改进懒加载和缓存机制
|
||||
- `apps/desktop/src/types/material.ts`
|
||||
- 添加新的音频API类型定义
|
||||
|
||||
## 使用示例
|
||||
|
||||
```tsx
|
||||
// 在MaterialCard或其他组件中使用
|
||||
<MaterialThumbnail
|
||||
material={material}
|
||||
size="medium"
|
||||
thumbnailCache={thumbnailCache}
|
||||
setThumbnailCache={setThumbnailCache}
|
||||
/>
|
||||
```
|
||||
|
||||
## 安全考虑
|
||||
|
||||
1. **文件访问安全**:所有文件访问都通过后端API,避免直接使用file://协议
|
||||
2. **路径清理**:自动处理Windows长路径前缀
|
||||
3. **文件存在性检查**:在访问文件前验证文件是否存在
|
||||
4. **错误处理**:完善的错误处理和用户反馈
|
||||
|
||||
## 性能优化
|
||||
|
||||
1. **懒加载**:只有当素材可见时才加载内容
|
||||
2. **缓存机制**:避免重复请求相同素材
|
||||
3. **异步处理**:所有文件操作都是异步的
|
||||
4. **内存管理**:音频元素的正确清理
|
||||
|
||||
## 测试建议
|
||||
|
||||
1. 测试不同格式的图片文件显示
|
||||
2. 测试视频缩略图生成
|
||||
3. 测试音频播放功能
|
||||
4. 测试文件不存在的错误处理
|
||||
5. 测试大文件的加载性能
|
||||
|
||||
## 后续改进
|
||||
|
||||
1. 可以考虑添加图片缩略图生成以提高加载速度
|
||||
2. 音频播放可以添加进度条和音量控制
|
||||
3. 可以添加更多音频格式支持
|
||||
4. 考虑添加视频预览播放功能
|
||||
|
|
@ -77,6 +77,7 @@ pub fn run() {
|
|||
commands::material_commands::read_thumbnail_as_data_url,
|
||||
commands::material_commands::get_material_thumbnail_base64,
|
||||
commands::material_commands::get_segment_thumbnail_base64,
|
||||
commands::material_commands::get_audio_file_base64,
|
||||
commands::material_commands::test_scene_detection,
|
||||
commands::material_commands::get_material_segments,
|
||||
commands::material_commands::get_material_segment_by_id,
|
||||
|
|
|
|||
|
|
@ -953,26 +953,75 @@ pub async fn get_material_thumbnail_base64(
|
|||
}
|
||||
}
|
||||
|
||||
// 缩略图不存在或文件已丢失,需要重新生成
|
||||
let video_path = &material.original_path;
|
||||
// 根据素材类型处理
|
||||
match material.material_type {
|
||||
crate::data::models::material::MaterialType::Image => {
|
||||
// 图片类型:直接读取图片文件并返回base64
|
||||
let image_path = &material.original_path;
|
||||
|
||||
// 去掉Windows长路径前缀
|
||||
let clean_video_path = if video_path.starts_with("\\\\?\\") {
|
||||
&video_path[4..]
|
||||
} else {
|
||||
video_path
|
||||
};
|
||||
// 去掉Windows长路径前缀
|
||||
let clean_image_path = if image_path.starts_with("\\\\?\\") {
|
||||
&image_path[4..]
|
||||
} else {
|
||||
image_path
|
||||
};
|
||||
|
||||
// 检查视频文件是否存在,不存在则报错
|
||||
if !Path::new(clean_video_path).exists() {
|
||||
let error_msg = format!("视频文件不存在,无法生成缩略图: {}", clean_video_path);
|
||||
tracing::error!(
|
||||
material_id = %material_id,
|
||||
video_path = %clean_video_path,
|
||||
"视频文件不存在"
|
||||
);
|
||||
return Err(error_msg);
|
||||
}
|
||||
// 检查图片文件是否存在
|
||||
if !Path::new(clean_image_path).exists() {
|
||||
let error_msg = format!("图片文件不存在: {}", clean_image_path);
|
||||
tracing::error!(
|
||||
material_id = %material_id,
|
||||
image_path = %clean_image_path,
|
||||
"图片文件不存在"
|
||||
);
|
||||
return Err(error_msg);
|
||||
}
|
||||
|
||||
// 直接读取图片文件
|
||||
let file_data = fs::read(clean_image_path)
|
||||
.map_err(|e| format!("读取图片文件失败: {}", e))?;
|
||||
|
||||
// 根据文件扩展名确定MIME类型
|
||||
let mime_type = match Path::new(clean_image_path)
|
||||
.extension()
|
||||
.and_then(|ext| ext.to_str())
|
||||
.map(|ext| ext.to_lowercase())
|
||||
.as_deref()
|
||||
{
|
||||
Some("jpg") | Some("jpeg") => "image/jpeg",
|
||||
Some("png") => "image/png",
|
||||
Some("gif") => "image/gif",
|
||||
Some("webp") => "image/webp",
|
||||
Some("bmp") => "image/bmp",
|
||||
Some("tiff") | Some("tif") => "image/tiff",
|
||||
Some("svg") => "image/svg+xml",
|
||||
_ => "image/jpeg", // 默认为JPEG
|
||||
};
|
||||
|
||||
let base64_data = general_purpose::STANDARD.encode(&file_data);
|
||||
return Ok(format!("data:{};base64,{}", mime_type, base64_data));
|
||||
}
|
||||
crate::data::models::material::MaterialType::Video => {
|
||||
// 视频类型:生成缩略图
|
||||
let video_path = &material.original_path;
|
||||
|
||||
// 去掉Windows长路径前缀
|
||||
let clean_video_path = if video_path.starts_with("\\\\?\\") {
|
||||
&video_path[4..]
|
||||
} else {
|
||||
video_path
|
||||
};
|
||||
|
||||
// 检查视频文件是否存在,不存在则报错
|
||||
if !Path::new(clean_video_path).exists() {
|
||||
let error_msg = format!("视频文件不存在,无法生成缩略图: {}", clean_video_path);
|
||||
tracing::error!(
|
||||
material_id = %material_id,
|
||||
video_path = %clean_video_path,
|
||||
"视频文件不存在"
|
||||
);
|
||||
return Err(error_msg);
|
||||
}
|
||||
|
||||
// 生成缩略图路径
|
||||
let thumbnail_filename = format!("{}_material_thumbnail.jpg", material.id);
|
||||
|
|
@ -1070,12 +1119,18 @@ pub async fn get_material_thumbnail_base64(
|
|||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
|
||||
// 读取生成的缩略图文件并返回base64
|
||||
let file_data = fs::read(&thumbnail_path_str)
|
||||
.map_err(|e| format!("读取生成的缩略图失败: {}", e))?;
|
||||
// 读取生成的缩略图文件并返回base64
|
||||
let file_data = fs::read(&thumbnail_path_str)
|
||||
.map_err(|e| format!("读取生成的缩略图失败: {}", e))?;
|
||||
|
||||
let base64_data = general_purpose::STANDARD.encode(&file_data);
|
||||
Ok(format!("data:image/jpeg;base64,{}", base64_data))
|
||||
let base64_data = general_purpose::STANDARD.encode(&file_data);
|
||||
Ok(format!("data:image/jpeg;base64,{}", base64_data))
|
||||
}
|
||||
_ => {
|
||||
// 其他类型(音频等)不支持缩略图
|
||||
Err(format!("不支持的素材类型: {:?}", material.material_type))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 根据segmentId获取缩略图base64数据URL
|
||||
|
|
@ -1268,6 +1323,80 @@ pub async fn get_segment_thumbnail_base64(
|
|||
Ok(format!("data:image/jpeg;base64,{}", base64_data))
|
||||
}
|
||||
|
||||
/// 获取音频文件的base64数据URL
|
||||
#[command]
|
||||
pub async fn get_audio_file_base64(
|
||||
material_id: String,
|
||||
app_state: State<'_, AppState>,
|
||||
) -> Result<String, String> {
|
||||
use std::fs;
|
||||
use base64::{Engine as _, engine::general_purpose};
|
||||
|
||||
// 获取素材信息
|
||||
let material = {
|
||||
let material_repository_guard = app_state.material_repository.lock().unwrap();
|
||||
let material_repository = material_repository_guard.as_ref()
|
||||
.ok_or("MaterialRepository未初始化")?;
|
||||
|
||||
material_repository.get_by_id(&material_id)
|
||||
.map_err(|e| e.to_string())?
|
||||
};
|
||||
|
||||
let material = match material {
|
||||
Some(m) => m,
|
||||
None => return Err("素材不存在".to_string()),
|
||||
};
|
||||
|
||||
// 检查是否为音频类型
|
||||
if !matches!(material.material_type, crate::data::models::material::MaterialType::Audio) {
|
||||
return Err("不是音频类型的素材".to_string());
|
||||
}
|
||||
|
||||
let audio_path = &material.original_path;
|
||||
|
||||
// 去掉Windows长路径前缀
|
||||
let clean_audio_path = if audio_path.starts_with("\\\\?\\") {
|
||||
&audio_path[4..]
|
||||
} else {
|
||||
audio_path
|
||||
};
|
||||
|
||||
// 检查音频文件是否存在
|
||||
if !std::path::Path::new(clean_audio_path).exists() {
|
||||
let error_msg = format!("音频文件不存在: {}", clean_audio_path);
|
||||
tracing::error!(
|
||||
material_id = %material_id,
|
||||
audio_path = %clean_audio_path,
|
||||
"音频文件不存在"
|
||||
);
|
||||
return Err(error_msg);
|
||||
}
|
||||
|
||||
// 读取音频文件
|
||||
let file_data = fs::read(clean_audio_path)
|
||||
.map_err(|e| format!("读取音频文件失败: {}", e))?;
|
||||
|
||||
// 根据文件扩展名确定MIME类型
|
||||
let mime_type = match std::path::Path::new(clean_audio_path)
|
||||
.extension()
|
||||
.and_then(|ext| ext.to_str())
|
||||
.map(|ext| ext.to_lowercase())
|
||||
.as_deref()
|
||||
{
|
||||
Some("mp3") => "audio/mpeg",
|
||||
Some("wav") => "audio/wav",
|
||||
Some("flac") => "audio/flac",
|
||||
Some("aac") => "audio/aac",
|
||||
Some("ogg") => "audio/ogg",
|
||||
Some("wma") => "audio/x-ms-wma",
|
||||
Some("m4a") => "audio/mp4",
|
||||
_ => "audio/mpeg", // 默认为MP3
|
||||
};
|
||||
|
||||
let base64_data = general_purpose::STANDARD.encode(&file_data);
|
||||
Ok(format!("data:{};base64,{}", mime_type, base64_data))
|
||||
}
|
||||
|
||||
/// 测试FFmpeg缩略图生成命令(用于调试)
|
||||
#[command]
|
||||
pub async fn test_ffmpeg_thumbnail(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import React, { useState, useEffect } from 'react';
|
||||
import { FileVideo, FileAudio, FileImage, File, Loader2 } from 'lucide-react';
|
||||
import { FileVideo, FileAudio, FileImage, File, Loader2, Play, Pause, Volume2 } from 'lucide-react';
|
||||
import { Material } from '../types/material';
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { useLazyLoad } from '../hooks/useLazyLoad';
|
||||
|
|
@ -27,6 +27,8 @@ export const MaterialThumbnail: React.FC<MaterialThumbnailProps> = ({
|
|||
const [loading, setLoading] = useState(false);
|
||||
const [thumbnailUrl, setThumbnailUrl] = useState<string | null>(null);
|
||||
const [error, setError] = useState(false);
|
||||
const [isPlaying, setIsPlaying] = useState(false);
|
||||
const [audioElement, setAudioElement] = useState<HTMLAudioElement | null>(null);
|
||||
|
||||
// 使用懒加载Hook,当缩略图容器可见时才开始加载
|
||||
const { isVisible, elementRef } = useLazyLoad(0.1, '100px');
|
||||
|
|
@ -61,14 +63,66 @@ export const MaterialThumbnail: React.FC<MaterialThumbnailProps> = ({
|
|||
}
|
||||
};
|
||||
|
||||
// 音频播放控制
|
||||
const toggleAudioPlay = async () => {
|
||||
if (material.material_type !== 'Audio') return;
|
||||
|
||||
if (!audioElement) {
|
||||
try {
|
||||
// 通过后端API获取音频数据
|
||||
console.log('获取音频数据:', material.id);
|
||||
const audioDataUrl = await invoke<string>('get_audio_file_base64', {
|
||||
materialId: material.id
|
||||
});
|
||||
console.log('获取音频数据成功');
|
||||
|
||||
// 创建音频元素
|
||||
const audio = new Audio();
|
||||
audio.src = audioDataUrl;
|
||||
audio.onended = () => setIsPlaying(false);
|
||||
audio.onerror = () => {
|
||||
console.error('音频播放失败');
|
||||
setIsPlaying(false);
|
||||
};
|
||||
setAudioElement(audio);
|
||||
|
||||
await audio.play();
|
||||
setIsPlaying(true);
|
||||
} catch (error) {
|
||||
console.error('音频播放失败:', error);
|
||||
setIsPlaying(false);
|
||||
}
|
||||
} else {
|
||||
if (isPlaying) {
|
||||
audioElement.pause();
|
||||
setIsPlaying(false);
|
||||
} else {
|
||||
try {
|
||||
await audioElement.play();
|
||||
setIsPlaying(true);
|
||||
} catch (error) {
|
||||
console.error('音频播放失败:', error);
|
||||
setIsPlaying(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// 清理音频元素
|
||||
useEffect(() => {
|
||||
// 只有当元素可见时才加载缩略图
|
||||
return () => {
|
||||
if (audioElement) {
|
||||
audioElement.pause();
|
||||
audioElement.src = '';
|
||||
}
|
||||
};
|
||||
}, [audioElement]);
|
||||
|
||||
useEffect(() => {
|
||||
// 只有当元素可见时才加载内容
|
||||
if (!isVisible) return;
|
||||
|
||||
// 只为视频类型生成缩略图
|
||||
if (material.material_type !== 'Video') return;
|
||||
|
||||
const loadThumbnail = async () => {
|
||||
const loadContent = async () => {
|
||||
const materialId = material.id;
|
||||
|
||||
// 检查缓存
|
||||
|
|
@ -78,32 +132,83 @@ export const MaterialThumbnail: React.FC<MaterialThumbnailProps> = ({
|
|||
return;
|
||||
}
|
||||
|
||||
// 加载缩略图
|
||||
setLoading(true);
|
||||
setError(false);
|
||||
|
||||
try {
|
||||
console.log('获取素材缩略图:', materialId);
|
||||
const dataUrl = await invoke<string>('get_material_thumbnail_base64', {
|
||||
materialId: materialId
|
||||
});
|
||||
console.log('获取缩略图成功');
|
||||
setThumbnailUrl(dataUrl);
|
||||
// 根据素材类型加载不同内容
|
||||
if (material.material_type === 'Video') {
|
||||
// 视频:生成缩略图
|
||||
setLoading(true);
|
||||
setError(false);
|
||||
|
||||
// 更新缓存
|
||||
const newCache = new Map(thumbnailCache);
|
||||
newCache.set(materialId, dataUrl);
|
||||
setThumbnailCache(newCache);
|
||||
} catch (error) {
|
||||
console.error('获取缩略图失败:', error);
|
||||
setError(true);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
try {
|
||||
console.log('获取视频缩略图:', materialId);
|
||||
const dataUrl = await invoke<string>('get_material_thumbnail_base64', {
|
||||
materialId: materialId
|
||||
});
|
||||
console.log('获取缩略图成功');
|
||||
setThumbnailUrl(dataUrl);
|
||||
|
||||
// 更新缓存
|
||||
const newCache = new Map(thumbnailCache);
|
||||
newCache.set(materialId, dataUrl);
|
||||
setThumbnailCache(newCache);
|
||||
} catch (error) {
|
||||
console.error('获取缩略图失败:', error);
|
||||
setError(true);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
} else if (material.material_type === 'Image') {
|
||||
// 图片:通过后端API获取base64数据
|
||||
setLoading(true);
|
||||
setError(false);
|
||||
|
||||
try {
|
||||
console.log('获取图片数据:', materialId);
|
||||
const dataUrl = await invoke<string>('get_material_thumbnail_base64', {
|
||||
materialId: materialId
|
||||
});
|
||||
console.log('获取图片数据成功');
|
||||
setThumbnailUrl(dataUrl);
|
||||
|
||||
// 更新缓存
|
||||
const newCache = new Map(thumbnailCache);
|
||||
newCache.set(materialId, dataUrl);
|
||||
setThumbnailCache(newCache);
|
||||
} catch (error) {
|
||||
console.error('获取图片数据失败:', error);
|
||||
setError(true);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}
|
||||
// 音频类型不需要加载缩略图,直接显示播放控件
|
||||
};
|
||||
|
||||
loadThumbnail();
|
||||
}, [isVisible, material.id, material.material_type, thumbnailCache, setThumbnailCache]);
|
||||
loadContent();
|
||||
}, [isVisible, material.id, material.material_type, material.original_path, thumbnailCache, setThumbnailCache]);
|
||||
|
||||
// 渲染音频播放控件
|
||||
const renderAudioPlayer = () => {
|
||||
const iconSize = size === 'small' ? 'w-4 h-4' : size === 'large' ? 'w-8 h-8' : 'w-6 h-6';
|
||||
|
||||
return (
|
||||
<div className="relative w-full h-full bg-gradient-to-br from-green-50 to-green-100 rounded-lg flex flex-col items-center justify-center cursor-pointer hover:from-green-100 hover:to-green-200 transition-colors"
|
||||
onClick={toggleAudioPlay}>
|
||||
<Volume2 className={`${iconSize} text-green-600 mb-1`} />
|
||||
<div className="flex items-center justify-center">
|
||||
{isPlaying ? (
|
||||
<Pause className={`${size === 'small' ? 'w-3 h-3' : size === 'large' ? 'w-5 h-5' : 'w-4 h-4'} text-green-700`} />
|
||||
) : (
|
||||
<Play className={`${size === 'small' ? 'w-3 h-3' : size === 'large' ? 'w-5 h-5' : 'w-4 h-4'} text-green-700`} />
|
||||
)}
|
||||
</div>
|
||||
{size !== 'small' && (
|
||||
<div className="text-xs text-green-600 mt-1 font-medium">
|
||||
{isPlaying ? '播放中' : '点击播放'}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
return (
|
||||
<div
|
||||
|
|
@ -112,10 +217,14 @@ export const MaterialThumbnail: React.FC<MaterialThumbnailProps> = ({
|
|||
>
|
||||
{loading ? (
|
||||
<Loader2 className={`${size === 'small' ? 'w-3 h-3' : size === 'large' ? 'w-6 h-6' : 'w-4 h-4'} animate-spin text-blue-600`} />
|
||||
) : material.material_type === 'Audio' ? (
|
||||
// 音频:显示播放控件
|
||||
renderAudioPlayer()
|
||||
) : thumbnailUrl && !error ? (
|
||||
// 图片和视频:显示图片/缩略图
|
||||
<img
|
||||
src={thumbnailUrl}
|
||||
alt={`${material.name} 缩略图`}
|
||||
alt={`${material.name} ${material.material_type === 'Image' ? '图片' : '缩略图'}`}
|
||||
className="w-full h-full object-cover rounded-lg"
|
||||
onError={() => {
|
||||
setError(true);
|
||||
|
|
@ -123,6 +232,7 @@ export const MaterialThumbnail: React.FC<MaterialThumbnailProps> = ({
|
|||
}}
|
||||
/>
|
||||
) : isVisible ? (
|
||||
// 加载失败或其他类型:显示类型图标
|
||||
getTypeIcon()
|
||||
) : (
|
||||
// 未加载时显示占位符
|
||||
|
|
|
|||
|
|
@ -178,6 +178,9 @@ export interface MaterialCommands {
|
|||
width: number,
|
||||
height: number
|
||||
): Promise<void>;
|
||||
|
||||
// 音频文件访问命令
|
||||
get_audio_file_base64(materialId: string): Promise<string>;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Reference in New Issue