怎样做国外网站推广,网页前端模板,单人做网站,长沙做网站的费用结合SK和ChatGLM3BwhisperAvalonia实现语音切换城市
先创建一个Avalonia的MVVM项目模板,项目名称GisApp
项目创建完成以后添加以下nuget依赖
PackageReference IncludeMapsui.Avalonia Version4.1.1 /
PackageReference IncludeM…结合SK和ChatGLM3BwhisperAvalonia实现语音切换城市
先创建一个Avalonia的MVVM项目模板,项目名称GisApp
项目创建完成以后添加以下nuget依赖
PackageReference IncludeMapsui.Avalonia Version4.1.1 /
PackageReference IncludeMicrosoft.Extensions.DependencyInjection Version8.0.0 /
PackageReference IncludeMicrosoft.Extensions.Http Version8.0.0 /
PackageReference IncludeMicrosoft.SemanticKernel Version1.0.0-beta8 /
PackageReference IncludeNAudio Version2.2.1 /
PackageReference IncludeWhisper.net Version1.5.0 /
PackageReference IncludeWhisper.net.Runtime Version1.5.0 /Mapsui.Avalonia是Avalonia的一个Gis地图组件Microsoft.Extensions.DependencyInjection用于构建一个DI容器Microsoft.Extensions.Http用于注册一个HttpClient工厂Microsoft.SemanticKernel则是SK用于构建AI插件NAudio是一个用于录制语音的工具包Whisper.net是一个.NET的Whisper封装Whisper用的是OpenAI开源的语音识别模型Whisper.net.Runtime属于Whisper
修改App.cs
打开App.cs修改成以下代码
public partial class App : Application
{public override void Initialize(){AvaloniaXamlLoader.Load(this);}public override void OnFrameworkInitializationCompleted(){if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop){var services new ServiceCollection();services.AddSingletonMainWindow((services) new MainWindow(services.GetRequiredServiceIKernel(), services.GetRequiredServiceWhisperProcessor()){DataContext new MainWindowViewModel(),});services.AddHttpClient();var openAIHttpClientHandler new OpenAIHttpClientHandler();var httpClient new HttpClient(openAIHttpClientHandler);services.AddTransientIKernel((serviceProvider) {return new KernelBuilder().WithOpenAIChatCompletionService(gpt-3.5-turbo-16k, fastgpt-zE0ub2ZxvPMwtd6XYgDX8jyn5ubiC,httpClient: httpClient).Build();});services.AddSingleton(() {var ggmlType GgmlType.Base;// 定义使用模型var modelFileName ggml-base.bin;return WhisperFactory.FromPath(modelFileName).CreateBuilder().WithLanguage(auto) // auto则是自动识别语言.Build();});var serviceProvider services.BuildServiceProvider();desktop.MainWindow serviceProvider.GetRequiredServiceMainWindow();}base.OnFrameworkInitializationCompleted();}
}OpenAIHttpClientHandler.cs这个文件是用于修改SK的访问地址默认的SK只支持OpenAI官方的地址并且不能进行修改
public class OpenAIHttpClientHandler : HttpClientHandler
{protected override TaskHttpResponseMessage SendAsync(HttpRequestMessage request, CancellationToken cancellationToken){if (request.RequestUri.LocalPath /v1/chat/completions){var uriBuilder new UriBuilder(http://您的ChatGLM3B地址/api/v1/chat/completions);request.RequestUri uriBuilder.Uri;}return base.SendAsync(request, cancellationToken);}
}修改ViewModels/MainWindowViewModel.cs
public class MainWindowViewModel : ViewModelBase
{private string subtitle string.Empty;public string Subtitle{get subtitle;set this.RaiseAndSetIfChanged(ref subtitle, value);}private Bitmap butBackground;public Bitmap ButBackground{get butBackground;set this.RaiseAndSetIfChanged(ref butBackground, value);}
}ButBackground是显示麦克风图标的写到模型是为了切换图标Subtitle用于显示识别的文字
添加SK插件
创建文件/plugins/MapPlugin/AcquireLatitudeLongitude/config.json这个是插件的相关配置信息
{schema: 1,type: completion,description: 获取坐标,completion: {max_tokens: 1000,temperature: 0.3,top_p: 0.0,presence_penalty: 0.0,frequency_penalty: 0.0},input: {parameters: [{name: input,description: 获取坐标,defaultValue: }]}
}创建文件/plugins/MapPlugin/AcquireLatitudeLongitude/skprompt.txt下面是插件的prompt通过以下内容可以提取用户城市然后得到城市的经纬度
请返回{{$input}}的经纬度然后返回以下格式不要回复只需要下面这个格式
{latitude:,longitude:
}修改Views/MainWindow.axaml代码将[素材](# 素材)添加到Assets中
Window xmlnshttps://github.com/avaloniauixmlns:xhttp://schemas.microsoft.com/winfx/2006/xamlxmlns:vmusing:GisApp.ViewModelsxmlns:dhttp://schemas.microsoft.com/expression/blend/2008xmlns:mchttp://schemas.openxmlformats.org/markup-compatibility/2006mc:Ignorabled d:DesignWidth800 d:DesignHeight450x:ClassGisApp.Views.MainWindowx:DataTypevm:MainWindowViewModelIcon/Assets/avalonia-logo.icoWidth800Height800TitleGisAppDesign.DataContextvm:MainWindowViewModel //Design.DataContextGridGrid NameMapStackPanel/GridStackPanel HorizontalAlignmentRight VerticalAlignmentBottom BackgroundTransparent Margin25TextBlock ForegroundBlack Text{Binding Subtitle} Width80 TextWrappingWrapWithOverflow Padding8/TextBlockButton Width60 ClickButton_OnClick BackgroundTransparent VerticalAlignmentCenter HorizontalAlignmentCenterImage NameButBackground Source{Binding ButBackground} Height40 Width40/Image/Button/StackPanel/Grid
/Window修改Views/MainWindow.axaml.cs代码
public partial class MainWindow : Window
{private bool openVoice false;private WaveInEvent waveIn;private readonly IKernel _kernel;private readonly WhisperProcessor _processor;private readonly Channelstring _channel Channel.CreateUnboundedstring();private MapControl mapControl;public MainWindow(IKernel kernel, WhisperProcessor processor){_kernel kernel;_processor processor;InitializeComponent();mapControl new MapControl();// 默认定位到深圳mapControl.Map new Map(){CRS EPSG:3857,Home n {var centerOfLondonOntario new MPoint(114.06667, 22.61667);var sphericalMercatorCoordinate SphericalMercator.FromLonLat(centerOfLondonOntario.X, centerOfLondonOntario.Y).ToMPoint();n.ZoomToLevel(15);n.CenterOnAndZoomTo(sphericalMercatorCoordinate, n.Resolutions[15]);}};mapControl.Map?.Layers.Add(Mapsui.Tiling.OpenStreetMap.CreateTileLayer());MapStackPanel.Children.Add(mapControl);DataContextChanged (sender, args) {using var voice AssetLoader.Open(new Uri(avares://GisApp/Assets/voice.png));ViewModel.ButBackground new Avalonia.Media.Imaging.Bitmap(voice);};Task.Factory.StartNew(ReadMessage);}private MainWindowViewModel ViewModel (MainWindowViewModel)DataContext;private void Button_OnClick(object? sender, RoutedEventArgs e){if (openVoice){using var voice AssetLoader.Open(new Uri(avares://GisApp/Assets/voice.png));ViewModel.ButBackground new Avalonia.Media.Imaging.Bitmap(voice);waveIn.StopRecording();}else{using var voice AssetLoader.Open(new Uri(avares://GisApp/Assets/open-voice.png));ViewModel.ButBackground new Avalonia.Media.Imaging.Bitmap(voice);// 获取当前麦克风设备waveIn new WaveInEvent();waveIn.DeviceNumber 0; // 选择麦克风设备0通常是默认设备WaveFileWriter writer new WaveFileWriter(recorded.wav, waveIn.WaveFormat);// 设置数据接收事件waveIn.DataAvailable (sender, a) {Console.WriteLine($接收到音频数据: {a.BytesRecorded} 字节);writer.Write(a.Buffer, 0, a.BytesRecorded);if (writer.Position waveIn.WaveFormat.AverageBytesPerSecond * 30){waveIn.StopRecording();}};// 录音结束事件waveIn.RecordingStopped async (sender, e) {writer?.Dispose();writer null;waveIn.Dispose();await using var fileStream File.OpenRead(recorded.wav);using var wavStream new MemoryStream();await using var reader new WaveFileReader(fileStream);var resampler new WdlResamplingSampleProvider(reader.ToSampleProvider(), 16000);WaveFileWriter.WriteWavFileToStream(wavStream, resampler.ToWaveProvider16());wavStream.Seek(0, SeekOrigin.Begin);await Dispatcher.UIThread.InvokeAsync(() { ViewModel.Subtitle string.Empty; });string text string.Empty;await foreach (var result in _processor.ProcessAsync(wavStream)){await Dispatcher.UIThread.InvokeAsync(() { ViewModel.Subtitle text result.Text; });}_channel.Writer.TryWrite(text);};Console.WriteLine(开始录音...);waveIn.StartRecording();}openVoice !openVoice;}private async Task ReadMessage(){try{var pluginsDirectory Path.Combine(Directory.GetCurrentDirectory(), plugins);var chatPlugin _kernel.ImportSemanticFunctionsFromDirectory(pluginsDirectory, MapPlugin);// 循环读取管道中的数据while (await _channel.Reader.WaitToReadAsync()){// 读取管道中的数据while (_channel.Reader.TryRead(out var message)){// 使用AcquireLatitudeLongitude插件解析用户输入的地点然后得到地点的经纬度var value await _kernel.RunAsync(new ContextVariables{[input] message}, chatPlugin[AcquireLatitudeLongitude]);// 解析字符串成模型var acquireLatitudeLongitude JsonSerializer.DeserializeAcquireLatitudeLongitude(value.ToString());// 使用MapPlugin插件定位到用户输入的地点var centerOfLondonOntario new MPoint(acquireLatitudeLongitude.longitude, acquireLatitudeLongitude.latitude);var sphericalMercatorCoordinate SphericalMercator.FromLonLat(centerOfLondonOntario.X, centerOfLondonOntario.Y).ToMPoint();// 默认使用15级缩放mapControl.Map.Navigator.ZoomToLevel(15);mapControl.Map.Navigator.CenterOnAndZoomTo(sphericalMercatorCoordinate, mapControl.Map.Navigator.Resolutions[15]);}}}catch (Exception e){Console.WriteLine(e);}}public class AcquireLatitudeLongitude{public double latitude { get; set; }public double longitude { get; set; }}
}流程讲解
用户点击了录制按钮触发了Button_OnClick事件然后在Button_OnClick事件中会打开用户的麦克风打开麦克风进行录制在录制结束事件中使用录制完成产生的wav文件然后拿到Whisper进行识别识别完成以后会将识别结果写入到_channelReadMessage则是一直监听_channel的数据当有数据写入这里则会读取到然后就将数据使用下面的sk执行AcquireLatitudeLongitude函数。 var value await _kernel.RunAsync(new ContextVariables{[input] message}, chatPlugin[AcquireLatitudeLongitude]);在解析value得到用户的城市经纬度通过mapControl.Map.Navigator修改到指定经纬度。
完整的操作流程就完成了当然实际业务会比这个更复杂。
素材 分享总结
讨论总结: 在本次会议中讨论了如何结合SK、ChatGLM3B、Whisper和Avalonia来实现语音切换城市的功能。具体讨论了创建Avalonia的MVVM项目模板添加了相关的NuGet依赖修改了App.cs、ViewModels/MainWindowViewModel.cs以及添加了SK插件的相关配置和文件。
行动项目:
创建Avalonia的MVVM项目模板项目名称为GisApp。添加所需的NuGet依赖包括Mapsui.Avalonia, Microsoft.Extensions.DependencyInjection, Microsoft.Extensions.Http, Microsoft.SemanticKernel, NAudio, Whisper.net和Whisper.net.Runtime。修改App.csOpenAIHttpClientHandler.csViewModels/MainWindowViewModel.cs以及相关的视图文件。添加SK插件包括创建相关的配置信息和prompt文件。实现录制语音、语音识别和切换城市的功能流程。
技术交流群737776595
