PDA

View Full Version : Just released: create your own stage notes with voice commands



mr_belowski
12-10-2020, 07:32 PM
This has just been released (15th October 2020) and opens up some interesting possiblities - there's nothing quite so arse-clenching as blasting over a flat out crest with only the pace notes you just created on a single sighting-run between you and certain death.

For Dirt Rally 2, first 2 minutes is me going the stage recce recording pace notes, then I restart the stage and run it in anger:

https://youtu.be/3fGTHlQbsmw

For Richard Burns Rally, this is a fast run after I created the pace notes

https://youtu.be/mn5KW9p9QEw

CC allows you to build your own stage / pace notes with voice commands during a stage recce run.

It works like this:
- start the stage with the app running in the right mode
- make the voice command "start stage reccy"
- when you pass a corner / obstacle, make a clear voice command. This is surprisingly hard, you only get 1 shot at it and the voice command must be delivered clearly and reasonably quickly. "left 3 don't cut keep in" would work, "left 3 don't cut" then a second later "keep in" might not.
- there needs to be at least half a second or so between voice commands.
- cruise the stage slowly doing all this - crests, bumps, ruts, fords, bridges, corners etc. You have to concentrate and it's easy to get muddled up. Remember to speak clearly and make each command an obvious single set of instructions. This is the hardest part, it takes practice
- if the app doesn't hear it'll tell you and you'll need to repeat the instruction so if you're going too fast this will be difficult / impossible to get the note in the right place
- if you make a call and it's clearly wrong (you screwed it up or it's been mis-recognised, you can replace the last call you recorded by saying "correction, [then the new correct call]"
- when you complete the stage make the command "finish stage reccy"once you've done that, you'll have a new file CrewChiefV4/RBR/[stage name]/pacenotes.json. You can inspect it or you can just tell the game to restart the stage immediately and see if you survive

pacenotes.json has each pace note along with the voice command that triggered its creation and any fragments of the voice command that the app wasn't able to process here's an example where I said "caution ford" but the app misheard is as "caution four"


{
"Distance": 1659.67273,
"Pacenote": "detail_caution",
"Modifier": "none",
"RawVoiceCommand": "caution four",
"UnprocessedVoiceCommandText": "four"
}


so rather than calling out "caution, ford" it just called "caution". Normally that last line is empty, and a single voice command may create more than one pace note - e.g.


{
"Distance": 1326.0437,
"Pacenote": "detail_caution",
"Modifier": "none",
"RawVoiceCommand": "caution square right don't cut"
},
{
"Distance": 1336.0437,
"Pacenote": "corner_square_right",
"Modifier": "detail_dont_cut",
"RawVoiceCommand": "caution square right don't cut"
}

One final note, for rbr the pace notes created with this mechanism will replace any that the game may be sending - if the app sees a pace notes file for the current stage it'll use this instead of the games own notes

Andreas Zetterlund
31-10-2020, 01:46 PM
Love the work you are doing! Pacenotes have been the biggest problem for me in this otherwise great game. I also use VR and since we can't use the pacenote plugin inside VR it's a hazzle to get good pacenotes.
Is it possible to get the CC overlay inside rbr? It is sometimes hard to get the correction, I ussualy get "matching partial response right", "Unable to find a pacenote to correct" so I have to say it a few times. Ussualy I'm sitting in the ditch after a off trying to correct the pacenote.

Other than that it feels very solid! A list of possible commands would be great also, is it possible to add both long and tightens to a corner?

mr_belowski
31-10-2020, 04:22 PM
Not sure why the CC overlay wouldn't work for you with RBR in VR - it works fine for me.

Regarding voice recognition, the corrections stuff is a bit more challenging than the initial pace notes creation.

The expectation is that it's a short command that only corrects a corner's tightness or its modifiers (tightens / long / cut / don't cut are the main ones). So a typical correction command would be made immediately after you pass (exit) a corner, and it would be simple like "correction, 4" or "correction, don't cut". The tightness (1, 2, 3, 4, 5, 6, flat, hairpin, square, k, medium, fast) can be combined with a modifier in a correction so "correction, 4, cut" should work. The app will apply the correction to the last corner it thinks you drove. If you also include a direction in the correction ("correction, right 3") then it will attempt to apply the correction to the most recently driven corner in the specified direction. If you're getting "unable to find pace note to correct" try making the correction without the direction.

The possible commands list is quite big. Most rally-type commands are supported. The best source for the list of commands is here in the source code:
https://gitlab.com/mr_belowski/CrewChiefV4/-/blob/master/CrewChiefV4/speech_recognition_config.txt

starting at line 450

it's still got some work left to do on it. For example, if you make a command "logs outside" there's no sound for this, so the app will just say "keep in". Same for "rocks inside" - it'll be called as "keep out".


these can be combined so you can have "left 3 long tightens then opens don't cut", or "caution flat right over crest cut" and stuff like that with a single command. There is a limit to the number of items a command can have - any more than 5 items in a command and the speech recogniser will start to get really unreliable.

Andreas Zetterlund
31-10-2020, 05:06 PM
Sorry I totally missed that there was an option for steamVR overlay. I'll give that a try tonight!

I tried it on a stage and the notes came out great on the first try, I had some troubles with threes and fours and I had to say them a few times to get it to recognize the note. This is probably because of my speech/accent as English is not my main language.

Is there a command to remove the previous note during recce? For example I had my "left 5 jump" become "left 4 jump", would it be possible to say "replace, left 5 jump" and it would replace the latest note with the new. This would remove the need for reversing back to get the timing correct also.

I was very impressed with the timing, how do you deal with that? Is just a flat distance or are you trying to figure out the speed carried through the corners and then figuring out how far in seconds to the next note?

I have been doing stage rally IRL for 2 years now so if you ever need some help or have any questions feel free, love the work you are doing. It's really great for the rally community and is making RBR much more accessible for more people.

mr_belowski
31-10-2020, 05:22 PM
3s and 4s can be replaced with "slow" and "medium", the speech recogniser doesn't care whether you use numbers or descriptions for corners and you can mix and match any combination of number, description, and direction first or direction last - the corner names the speech recogniser understands are

"1" / "acute"
"2" / "k"
"3" / "slow"
"4" / "medium"
"5" / "fast"
"6" / "slight"
"flat"
"square"

so "4 left", "left 4", "medium left" and "left medium" should all work the same.

if you make a pace note and need to correct it whilst doing a recce, you use "correction, left 5 jump". This deletes the most recently added pace note and replaces it with the one after "correction". It'll keep the distance data from the pace note it just deleted so in your example the "left 5 jump" will replace the "left 4 jump" and will be called at the same distance that the original note would have been called, even if you've travelled a bit before you made the correction command. Obviously this isn't perfect and if your recce is driven quickly and you need to make corrections it can get messy.


The timing uses some simple corner / obstacle length estimates based on the pace note you just created. It assumes that you've just passed the corner / obstacle when you make the recce instruction so it subtracts some distance from your current lap distance, taking into account things like corner speed, 'long' modifier and stuff, then estimates where the obstactle / corner might have started and uses that for the pace note's distance. The app also adds a separate 'distance to next corner / obstacle' placeholder pace note at the actual distance where you made the instruction and (depending on a few things) it might call a distance pace note at using this point.

We then use the car's speed to decide when to play each pace note using a look-ahead estimate (this is configurable, and can be adjusted on the fly during a stage with the "earlier calls" / "later calls" commands). The app also uses the car's speed and the number of calls it needs to read out to decide whether it should use the relaxed or the rushed versions of the sounds to give a sense of urgency where needed. I like the idea of this feature but I'm not sure it sounds quite 'right' yet.


Glad it's working well for you. It's a complicated piece of functionality and it's very hard to get right. It works great for me and I'm way faster through a stage using my own pace notes :). We're always open to feedback so will welcome any suggestions

Andreas Zetterlund
31-10-2020, 06:30 PM
One thing I would like to be added after doing a few stages now is to get feedback if the correction has registered. And maybe have two different recce modes. So currently the recce mode is great, but then what I'll would like to do is run the stage again, slowly and correct all errors while the codriver is reading the notes to me. I could do that by running the stage normally but I get no feedback that the correction has registered. So what I suggest is have another mode where he reads out the correction you just made for when you are driving the stage slowly correcting the pacenotes, and the other mode is when you are running the stage at speed then maybe you only get a "Got it" as acknowledgement as you still want to hear the rest of the notes.

EDIT: As a workout for now is it possible to get the console text big and clear in the steamOverlay? I can't seem to find a good option for this.
EDIT2: I manage to get the steamoverlay showing the entire CC app but unfortunately the app console does not scroll by itself to the latest message :(

Andreas Zetterlund
31-10-2020, 07:07 PM
A problem I just noticed is that none of the stages I made have been saved unfortunately. I could not find the .json files anywhere.

mr_belowski
31-10-2020, 07:36 PM
Did you make the "finish stage recce" command?

Andreas Zetterlund
31-10-2020, 07:44 PM
Ya and I got the "Playing sounds, events: codriver/acknowledge_end_recce". But there is no file, I also went into both RBR folder and CCV4 folder and make they all read and write thinking that it might be a admin/rights problem but no success so far.
Also I notice only saying correction and nothing after removes the pacenote, is that an intended feature? I said correction but forgot what note I was going to say and it deleted the old note, maybe something like "correction remove note" would be better for deleting notes.

mr_belowski
31-10-2020, 07:49 PM
So do none of your recce pace notes save?

Andreas Zetterlund
31-10-2020, 07:58 PM
Nope, I cant find the the folder mentioned even,CrewChiefV4/RBR/[stage name]/pacenotes.json. I checked both RBR folder and CC4 folder, not sure which one they should be saved in but there are none in any of the folders.

mr_belowski
31-10-2020, 08:23 PM
what stages are these? I've tested with the original stages and some of the RX plugin stages, both worked. I've just tested and it's working for me. Is there anything unusual about your Windows installation or location of your Documents folder? Any errors in the console window when you make the "finish stage recce" command?

[edit]

just to be clear, the pace notes json is in C:\Users\[your user name]\Documents\CrewChiefV4\RBR\[stage name]\

Andreas Zetterlund
31-10-2020, 08:37 PM
I found the folder now! Thanks, but I only have 1 folder in there called Rally HQ. I have been using the rbrTM/rbrcz tracks that comes with the trackpackage.
I have tried Kuadonvaara and Mitterbach so far, included in this list https://rbr.onlineracing.cz/index.php?act=rbrtracks

I just tried making notes at East west, an orginal stage and it worked. It saves pacenotes only for the orginal tracks.

mr_belowski
31-10-2020, 08:42 PM
the stages accessed from the RX plugin also work. I don't have the rbrcz track package so haven't tested that

mr_belowski
31-10-2020, 08:46 PM
also, do you have the latest version of the CrewChief plugin (CrewChief.dll) in the C:\RBR\Plugins folder? It should be exactly 142,848 bytes long, modified 24 ?September ?2020. If not, copy this file from C:\Program Files (x86)\Britton IT Ltd\CrewChiefV4\plugins\RBR\Plugins

Andreas Zetterlund
31-10-2020, 08:48 PM
Is there something I could try to make it be able to save the rbrtm tracks? I ussualy only do these since they are online.

mr_belowski
31-10-2020, 08:51 PM
did you check that the CC plugin is up to date?

Andreas Zetterlund
31-10-2020, 09:02 PM
Ya I'm running 4.14.0.3

mr_belowski
31-10-2020, 09:04 PM
no, i mean the Crew Chief plugin in the RBR/Plugins folder

Andreas Zetterlund
31-10-2020, 09:12 PM
Monday 26 october the CrewChief.dll was latest updated, I don't know how to check the version tho

mr_belowski
31-10-2020, 09:18 PM
Check the file size, or just copy the DLL from the cc install Plugins folder and re-test

Andreas Zetterlund
31-10-2020, 10:02 PM
I noticed in the log it says the track name is Rally HQ at one point but at the other it actually regonizes the name of the stage which is Karlstad. Does this help anything in debuging? https://imgur.com/a/q4hKg8W

I replaced the .dll into the plugins rbr folder. But nothing happend. The .dll is 140 kb, is this correct?

Edit: Realised I could just print the console here.
This is the console from a RBRTM stage, it seems to find the rbr trackid and rbr trackname, but it still loads it as Rally HQ.
New session, trigger data:
21:58:30.653 : SessionType: Race
21:58:30.653 : SessionPhase: Countdown
21:58:30.653 : SessionNumberOfLaps: 1
21:58:30.653 : EventIndex: 0
21:58:30.653 : RBR CarID: 3
21:58:30.653 : RBR TrackID: 481
21:58:30.653 : RBR TrackName: Karlstad
21:58:30.653 : RBR TrackCountry: Sweeden
21:58:30.653 : RBR TrackSurface: Snow
21:58:30.653 : RBR Approximate length: 1.900km
21:58:30.660 : Single-Class event:"" (unmapped)
21:58:30.667 : New session
21:58:30.667 : New session details...
21:58:30.667 : SessionType: Race
21:58:30.667 : EventIndex: 0
21:58:30.667 : SessionIteration: 0
21:58:30.667 : TrackName: "Rally HQ"
21:58:30.667 : Clearing game state...
21:58:30.668 : Purging regular queue
21:58:30.668 : Purging immediate queue
21:58:30.781 : LoadPacenotes: loaded 58 pacenotes with 14 unrecognised pacenotes


This is how it looks with a orginal/default stage.

22:15:23.673 : New session, trigger data:
22:15:23.673 : SessionType: Race
22:15:23.673 : SessionPhase: Countdown
22:15:23.673 : SessionNumberOfLaps: 1
22:15:23.673 : EventIndex: 0
22:15:23.673 : RBR CarID: 1
22:15:23.673 : RBR TrackID: 21
22:15:23.673 : RBR TrackName: Falstone
22:15:23.673 : RBR TrackCountry: Great Britain
22:15:23.673 : RBR TrackSurface: Gravel
22:15:23.673 : RBR Approximate length: 6.600km
22:15:23.673 : Single-Class event:"" (unmapped)
22:15:23.673 : New session
22:15:23.673 : New session details...
22:15:23.673 : SessionType: Race
22:15:23.673 : EventIndex: 0
22:15:23.673 : SessionIteration: 0
22:15:23.673 : TrackName: "Falstone"
22:15:23.673 : Clearing game state...
22:15:23.673 : Purging regular queue
22:15:23.673 : Purging immediate queue
22:15:23.774 : LoadPacenotes: loaded 141 pacenotes with 0 unrecognised pacenotes
22:16:05.287 : System recogniser recognised : "I'm OK", Confidence = 0.602
22:16:05.799 : Restarting speech recognition
22:16:05.799 : Opened channel - waiting for speech
22:16:05.799 : Getting audio from default device
22:16:31.587 : System recogniser recognised : "fine", Confidence = 0.749
22:16:32.099 : Restarting speech recognition
22:16:32.099 : Opened channel - waiting for speech
22:16:32.099 : Getting audio from default device

mr_belowski
31-10-2020, 10:59 PM
OK... there's something clearly not right here. The game is sending the correct track ID, which the Crew Chief code understands and we have this track ID in our tracks lookup. However, it's not setting the value in the track name field that our app is using. This isn't a bug in CC - CC is reading the correct track name value supplied by the game, but this value isn't being set properly. Perhaps it's an issue with some plugin(s) you're using in RBR.

Regardless of why it's happening, the CC can be altered to work around the issue - if CC recognises the track ID and the track name is set to "Rally HQ", then CC can ignore the nonsense name and use the name from the track it's got by looking up the track ID. This is a nasty hack - if the game sends some other nonsense track name it'll still cause issues.

Note that we need to use the track name field (rather than the name of the track we obtained by looking up the track ID) because some BTB tracks don't have a unique track ID.


So, to sum up, I think this is a bug in some RBR plugin but CC can be made to work around it, and this work around will be in the next CC version :)

Andreas Zetterlund
31-10-2020, 11:11 PM
Great to hear, I tried removing plugins such as the pacenote plugin but no success, it still says rally HQ.
Can't wait for the new version, thanks for the work this has been a lifesaver.

Andreas Zetterlund
01-11-2020, 11:24 AM
I thought I would summarize my feedback.
Bugs:
- Correction only works the second time, even if I say the exact words "Correction, right 2" it will not find the pacenote untill I say the exact same "correction, right 2" right after again. It's consistent atleast so I have to correct the corner twice for it to register.
- "Opens" is called as "wide out" even tho the pacenote "Opens" exist since I am using it for "Opens and tights", there it actually says Opens. But if you only add Opens it calls "wide out"
- RBRTM tracks not saving
- RBRTM pacenotes not reading correctly(for same reason as above)

Possible improvement:
- Another reccy mode, called correction reccy where he reads the pacenotes for you while you are driving the stage but when you correct a note he reads out "right 4 corrected to right 5" for example, so you get more detailed info about the correction.
- Possiblity to move a note later or sooner, sometimes I have added a note in the wrong place. I could edit them in the .json but having the possiblity to change it during correction reccy would be great. For example "correction, earlier" would push the note 0.5s further back, or you could manually say "correction earlier 1 second".
- Distance left, the possiblity to ask the co-driver how much there is left of a stage.
- Acknowledgement that co-driver has registered my correction during a stage

I find myself sometimes struggle with timing in very tight corners such as hairpins & 1s that he says the next corner after the hairpin before I even started braking. Is it possible to make him a bit later in the slower corners? I see there are some parameters but I'm unsure which one to correct.

Most of these points I have already covered in my previous posts but I thought I would summarize it for you so it's easier to see it all.
Thanks!

Edit: Possible to add this track workaround as a hotfix update? Would love to use CC pacenotes in the next weeks rally :D

mr_belowski
01-11-2020, 12:04 PM
is the correction issue during the recce or when you're actually driving the stage? Will see if i can repro that.

will look at 'opens', I seem to remember I had to hack this a little, might be able to fix that properly

RBRTM - will do a hotfix for this, probably today


"Correction, earlier" is implemented but not in recce mode. It moves the pace note back or forward 50 metres. I'll have a think about how this might be improved and will add an ack to the correction - just an "OK" is probably enough in rally mode.

tight corners is a problem I've encountered too. The look-ahead uses the car's current speed to decide which calls need to be read out and before you brake into the hairpin, you're going fast but the app doesn't take into account the amount you'll have to slow down. I did look at fixing this by implementing some expected speed adjustments but had to move to other stuff, it might be fixable.

Andreas Zetterlund
01-11-2020, 12:35 PM
This correction issue is only when driving the stage, it works fine when in recce mode.

I have never used "Correction, earlier" in recce mode and I don't see the need for it since you are not being read the pacenote so you don't actually know how it will be, so having it only in stage/recce correction mode is fine I would say.
Adding just an "OK" in stage mode is fine since you don't want to miss the coming notes so the less he speaks the better at that point.

The slow corners is not a huge dealbreaker since you are already going so slow that you probably don't even have enough speed for the next corner to matter that much. So ya it would be a nice feature down the line but I wouldn't say it's ruining anything. If you got good memory then it's fine hehe.

Great to hear about the hotfix! Can't wait :D

Another "problem" I have encountered is with long corners, is that because they are so long the co-drivers ussualy tell you a distance from the apex to the next corners. So say a we have "right 3 long into left 3" as a sequence of corners then he would say "right 3 long 120 left 3" since the distance from where I put the marker to the next corner is so long even if the right 3 goes directly into left 3.
I'm not even sure if and how this would be fixed. Maybe the possibility to add into as a note then it removes the distance between the note and overrides it with into? Not entirely sure what's the best way to go about this. But this is just a small detail I have noticed that can throw you off sometimes.

mr_belowski
01-11-2020, 02:27 PM
here's a beta with the work-around for the missing stage name

http://crewchief.isnais.de/CrewChiefV4-4.14.0.4_beta.msi

this also includes some changes I've not been able to properly test. The corrections stuff should work better (there should be no need to make a correction twice), the 'opens' modifier should work, and I've implemented some very basic 'pause' logic after calling a tight corner. After calling 'hairpin' (or a slow corner) the app will delay subsequent calls for between 1.5 and 4 seconds (depending on the length and tightness of the corner). Pace notes at the same distance as the corner aren't delayed (e.g it'll still play 'narrows' immediately if the narrows note is part of a 'hairpin left narrows' call). I don't know if this'll work properly - i've been unable to test (no time). If it's not right you can disable it with the "Pause co-driver after a slow corner" property.

Andreas Zetterlund
01-11-2020, 05:01 PM
Thanks! I just gave it a try and everything you mentioned works! I get an "okay" during stage when doing an correction, Opens now works also, correction works first time,
With the delay in hairpins I diden't notice anything to drastic but I did have some weird delays in the middle of stages, but this needs some more testing before I can be 100% certain but it felt like he called some of my 4s and 3s very late also. But I changed this with "correction, earlier" which by the way works great! Does it work to say "correction, earlier" twice to move the same note even further earlier? Or will it jump to the note before?

Also with saying only "correction" I think it removes the previous note, is this intended?

mr_belowski
01-11-2020, 06:42 PM
I think I'll revert the hairpins delay change. I did some brief testing and it's just not right - it's either too subtle to improve things or it delays too much and creates its own problems. I'll have to implement a better solution.

Saying correction only will delete a pace note. This is a bug, I'll fix it


Not sure what saying "correction, earlier" twice will do. I expect it'll move the same note twice, as long as you've not reached the next note

Andreas Zetterlund
02-11-2020, 08:33 AM
Just disabling "Pause co-driver after a slow corner" property would revert back to the old timings for me right? Using the beta version.

mr_belowski
02-11-2020, 08:42 AM
Yes, that's correct

Rick Bamford
06-11-2020, 10:25 AM
Hi All, Just downloaded CC and can't wait to give it a try. In the videos I've seen it looks absolutely what I have been looking for since starting rally games.

One thing I am trying to figure out though is how to create my own sound files as there are details in the default notes that are missing for my system. Or can I just request a list of terms to be included and wait?

mr_belowski
06-11-2020, 12:43 PM
if there are note items you need, give me a shout and i'll do the recordings. As each note item needs to be added to the speech recogniser, there's a balance to be struck between a rich range of note types and putting too much pressure on the recogniser (more possibilities means less accuracy). But it there's stuff we clearly need, i'll add it

ctbr
06-11-2020, 03:11 PM
So far I have made notes to a couple of stages and so far I miss "very long" and "go straight"
Very long needs no explanation and "go straight" I needed it for a stage that started with a straight and I want:
GO
go straight
400
hairpin left

Rick Bamford
07-11-2020, 05:36 AM
AH, Great, at least I haven't missed something. Just went through the list of notes and I have to admit there are more there than I realised at first glance. But here are the notes I am missing from my RL system.

Corners : This one isn't necessary but it would be good.
Open Hairpin

Corner Modifiers : These ones are extremely necessary for me
Plus - (1 right plus = Driven on throttle)
Minus - (1 Right Minus = Driven off Throttle)
And/On - (1 right on Crest = Corner happens at the same time as crest)
Into - (I know you have this set to some CrewChief wizardry and great work, but I really think it should be my call when it's put in not the co-drivers)
Then - (Crest Then 1 Right = Corner isn't straight after the crest but you need to position the car accordingly)

Also, Can you explain how to turn off the distance calls, I've never needed them and they are simply a distraction to me.

ok, thats all:)

mr_belowski
07-11-2020, 09:38 AM
"go straight" - can be added, this will also be useful for junction calls ("past junction, go straight")
"very long" - can be added
"open hairpin" - can be added

The minus / plus additional corner modifiers, i'll have to think about it. The issue is how they'd be stacked up with other modifiers.
"And / On" - hmm, is "right 3 over crest" unclear?

Changing the "into" behaviour might be possible. There's a 'Link codriver calls closer than this (metres)' property - if this is zero then CC won't do its own 'into' calls. But a code change would be needed to allow 'into' to be added manually.
Adding "then" would also need a code change.

Distance calls can be manually removed from the pacenotes json, there's currently no way to tune or disable them but that would be simple enough to add with a property to tune this behaviour and disable it.


While all the above sounds straightforward (and is, as far as the code is concerned) I think some of it could have a major impact on the performance of the speech recogniser, which would have to listen for a lot more combinations of words to include the 'into' / 'plus' / 'minus' / 'and' / 'on' and 'then'.

mr_belowski
08-11-2020, 10:31 AM
next release will have "open hairpin" corner and "go straight" as well as "very long" / "plus" / "minus" modifiers.

I've also made the auto-distance call configurable - this is the min-distance-to-next-pacenote before an auto generated distance call is inserted. The default is 40 metres, so if the next corner or obstacle call is 40 or more metres away the app may insert a distance pace note. If you want to disable the distance calls entirely then you can just set this to a very large number.

Not sure when the next release will be - I need to do some additional recordings and some testing (which also includes testing the dynamic look ahead changes to reduce call stacking after a very slow corner). The rest of the CC team are also very busy cooking up some new awesome so we might be delayed by that. I'll probably put out a beta version with these changes soon

Rick Bamford
11-11-2020, 11:41 AM
Thank you Mr belowski, the plus/minus thing will help me greatly and I look forward to trying it out. I have no idea what a voice recognition system can cope with, take your time and get it right, we can wait:).

Now, this part is just me explaining myself not expecting you to make changes (but if you can make it work then YAY) and I am not being aggressive, please don't take it as abuse lol.

If the "over crest" call is right for you (and I guess many others) then that's great and I'll have to get used to it. However, I had this discussion with my real life co-driver 20 years ago and I convinced him to go with "and". If you say the house is over the river do you mean it is on the water or on the other side of the river? To me "3 right over crest" is the same as saying "crest into 3 right". And the order of the call can be "crest and 3 right" or "3 right and crest", it depends on what the driver sees first, the crest or the corner. It is a matter of keeping the flow of the notes the same as the flow of the road.

Ok, off my soapbox now lol.

Andreas Zetterlund
11-11-2020, 12:31 PM
Hi again!

I've been running quite a few rallies with the beta version you uploaded, and on some stages it seems like certain notes and parts of the calls are missing. I will have to some more investigation but could this be because of the beta version?

Also when is the note placed? Is it the second I start speaking or is it when I finished my sentence? I have had occasions when some notes are placed 100-200m later then. For example I try to start my sentence in the apex or slightly before the apex of the corner(for hairpins I say it even before the corner to make sure it's early), but sometimes I've had the note placed 100m further down the road after the corner.

With the addtional pacenotes call, I agree that most of the modifiers that Rick Bamford mention are great to have! About the over/and discussion I have always used over or occasionally past.
For example "Right 5 over jump into left 3 past house" would mean that the right 5 starts and goes over a crest and then continues after the crest and then goes into a left 3 that is around a house for example. The past part is not very necessary for these kind of notes since I try to use them as a way of making sure we are in the right part of the notes. And also it sounds much better in Swedish then in English(I only use Swedish pacenotes irl).

Another pacenote that possibly could be used is "kinks" which just means that there are tons of very small corners that mostly can just be straightlined. This will also help with removing some work for the CC app, since I don't have to add all these 6 rights/lefts and he can focus on the tighter calls.

I was looking around the properties and the Default look ahead for codriver calls is set to 4 as default, and the rushed version is set to 2. Is this not suppose to be the opposite or have I missunderstood?

Say we are going at extremely high speeds and I need corners very fast, then I would need the corners much earlier than I would if we are doing hairpin after hairpins? So if I use the rushed codriver he will only say the note when it's 2 seconds ahead which means I only got 2s to react when I'm doing very fast speeds, while with the relaxed codriver I have 4 seconds after doing a hairpin?

I am probably incorrect about this part so please let me know how the parameters work.


EDIT: I noticed you already added the plus and many of the other modifiers while I was writing this post, great update I'll give it a try tonight!

mr_belowski
11-11-2020, 01:20 PM
I'll review the use of 'over' and think about it. It's kind of a catch-all for "drive over a crest" but I guess it's quite personal as to whether that feels right or not.

Regarding missing notes, this may be a bug or something that's not correctly wired up (for some modifiers or non-core notes), but if a corner call is missing entirely then it's clearly very serious. I've not encountered this myself but will keep an eye out for it.

Call placement is difficult to get right. The app expects the call to be created (in recce mode) at the end of a corner or obstacle (because you won't know if that 'long 4 left' should have 'tightens' until you actually complete the corner). This feels sensible to me because it allows the corner and it's modifier(s) (tightens, narrows, etc) and associated additional notes (rocks outside, etc) to be logged at the same time and allows you to create the call when you know all the details of the corner. However, the disadvantage of this is that the app has to modify the pace note's location to account for the obstacle / corner length in order to derive a sensible trigger point. We want to put the trigger point at the start of the corner but we can't create the pace note until the end of the corner. So the app has to estimate. It uses some finger-in-air values for how long a corner is (taking into account 'long' / 'very long' modifiers and a few other things). It's not perfect but it really shouldn't need to be - as long as the corner start point is approximately correct any error in the call timing should be small compared to the look-ahead time.

When you make a single voice command this command can contain a corner, a couple of modifiers and a couple of additional notes. For example, "right 3" [the corner]; "tightens", "don't cut" [the modifiers]; "rocks outside", "caution" [the additional notes]. In this example the app creates 1 entry for the corner and its modifiers, and one entry each for the additional notes. The app uses the stage distance at the point where you pressed the radio button for all of these notes to allow it to group them together so they are all called at the same trigger point. If you run the app with 'always on' voice recognition the app doesn't know when the voice command started so it has to use your stage distance at the point where it acknowledges the command. For this reason, it's far far better to use 'push and hold' voice recognition mode. If you must use 'always on' then you need to do your recce much slower so the pace note trigger point isn't too far from where you started your command.

The app should never move a call down the road but there may be cases where a particularly long corner gets called a bit late, if you create the pace note on the exit of that corner. I.e. an unusually long fast corner might be 300 metres long, you make the call at the end of the corner, the app assumes that 'very long right 6' corners are 200 metres long, so the app places the pace note trigger point 100m into the corner. In most cases like this being late shouldn't have a huge impact - if you're travelling 50m/s the call would be 2s late, and with a look ahead of 4 seconds it should be OK(ish). But it'll never be perfect - I could make the corner length estimates more generous but the downside of that is that calls will be earlier. When I'm doing a recce I wait till the end of a corner before making the pace note command but in cases where the corner is really really long I make the command earlier because i know that the corner start estimate will be a bit off.

The default look ahead is 4 seconds, for rushing there are 2 parameters - speed ('Codriver rushing speed threshold') and distance ('Codriver rushing look ahead'). The speed param means "always use rushed calls when going faster than this". The look ahead param means "use rushed calls if there's another batch of notes coming up in the next X seconds". The idea here is that we rush in cases where the notes are coming thick and fast, and in cases where the codriver is shitting himself because we're going a million miles an hour. So the default values do make sense and they work correctly (but I have to be honest, I don't like how the rushed calls sound so I change the 'Codriver rushing look ahead' value to be 0 to disable that behaviour).


Andreas - the release version is definitely worth trying - it's generally better than the beta. There's an additional property in there which adjusts the look-ahead time a little to account for a corner's expected speed. The goal is to delay calls about stuff after a very slow corner. Without this, the app uses the current speed to decide which notes need to be played. If we're approaching a slow corner but are still going fast, the app may decide it needs to tell the driver about lots of stuff after the slow corner, before he reaches that corner. This isn't necessarily what you want so this option ('Dynamic co-driver distance look ahead', enabled by default) should help a little here.

Andreas Zetterlund
15-11-2020, 08:50 AM
It works great with the new version!
I'm realized that I am missing a note, I would like to have for example.
"Right 5 tightens 2", "right 2 opens and tightens 1", having a number after the tightens so I know how much it tightens.
Other than that it has become apparent that the pacenotes now have become so good that it's my own note making skills that are the weakest point of the entire system haha.
Great job!

mr_belowski
15-11-2020, 08:53 AM
Yes, the "X tightens to Y" notes are a pretty glaring omission. I avoided them because I'd assumed they added a load of complications but perhaps it's not as bad as I'd feared. I'll do some digging. Maybe it would be safe to add a limited selection of all the possibilities or something. Will report back

Andreas Zetterlund
15-11-2020, 09:17 AM
I have sometimes used very long notes such as "Right 5 tightens bad over crest into gravel keep in" and the voice detection has no problems with it, gets it 100% of the time so I'm really impressed by the voice recognition. So I don't believe that "right 5 tightens 4" would be any problem for the program since it can handle my very long notes otherwise hehe.

mr_belowski
15-11-2020, 09:54 AM
the challenge is what to do with the corner modifiers. The implementation assumes that a command can contain zero or one corners, so any modifiers are applied to the corner (if it's part of the command). So the command
"right 4 opens, tightens to 2" would be interpreted correctly to 2 separate notes - 'right4+opens' and 'tightens2'. However, the command "right 4 tightens to 2, opens" would be incorrectly parsed. The "opens" modifier would be applied to the initial corner like the previous example so it would be parsed to the same calls - 'rght4+opens' and 'tightens2'. Which might end badly.

I think I have a way to make this work and place the modifiers in correct calls but i'm getting right to the limit of what the current parsers can handle without pulling it apart and re-doing it.

The other issue is that these tightens calls naturally encourage the use of connecting words like "and" / "then". These aren't handled at all by the speech recogniser - they're not part of its grammar so would interfere with recognition and would not be parsed. I'll think on that

mr_belowski
15-11-2020, 01:01 PM
OK, I think I've managed to crack this. I've reworked the parser a bit as part of some ongoing work to stop the app reordering elements of a call and added some more stuff. So now the app correctly recognises and parses commands like

"right four over crest don't cut and bridge then tightens to three"

"and" and "then" are, in this case, separate pace note calls so this parses into
right4+don't cut
over crest
and
bridge
then
tightens to three

It should put the modifiers on the right note now too

Andreas Zetterlund
15-11-2020, 03:10 PM
That's great to hear! So then or and needs to be called for when I want to link corners into one? How much can I stretch this? Say for example "Right 5 tightens 4 tightens 3"? Could I get this call by saying "Right 5 then tightens 4 then tightens 3"?

mr_belowski
15-11-2020, 04:11 PM
yes, that should work. I've just tested "right four over crest don't cut tightens to three then left five tightens to two", which works, as does "right four over crest don't cut tightens to three then tightens to two". Bear in mind that doing all this as a single command will result in each item having the same trigger distance - they'll be read togther.

When I say "works", I mean it's parsed correctly when I give the parser the text and it generates the expected pace note entries in the correct order. I need to do actual testing to confirm that the speech recogniser works correctly and that there are no nasty surprises. There's a limit of 8 items in a single command, where 1 item can be a corner; a 'tightens to'; a modifier; a link word (and/then/into); or an obstacle (crest, rocks outside, etc). So there's a limit to how much you can stack into a single command.

"Left four don't cut rocks inside tightens to three then ford and right two bad camber" works, but any more items would be missed off. But I'm way past what the parser was written to handle. I'm not expecting any issues with the speech recogniser here except a possible timeout (it takes a while to make the command)

mr_belowski
15-11-2020, 04:46 PM
ok.. further tests show the speech recogniser can't really handle this. The parser and logic works correctly but the speech recogniser gives up after a few seconds. The example of "Left four don't cut rocks inside tightens to three then ford and right two bad camber" works up to the end, where "right two bad camber" is mis-recognised (often as "right entry chicane" on my system). I'm seeing the speech recogniser stop listening and kick off the processing / parsing work while I'm still talking as well, if the command is very long.

So there's a practical limit here to what the recogniser can cope with but it's quite a lot already. I think it's unlikely you'd reach this limit unless you're deliberately testing unrealistic edge-cases like this. So it's looking promising. With any luck the changes will be in the next build, it feels like it's a major improvement.

One outstanding question is what to do with "caution" / "danger" items. Currently these are a special case and are always inserted at the front of a batch regardless of where they appear. So "right three caution rocks outside" would be read as "caution, right three rocks outside". I'm thinking this behaviour is probably not what's wanted and I should take it out and leave the caution / danger in the batch at the position it was inserted

Andreas Zetterlund
16-11-2020, 08:33 AM
I totally agree with the caution, especially now that you can have several turns in a note that the caution is not placed in front of everything. Will be interesting to see what can be done with the new version

mr_belowski
16-11-2020, 09:40 AM
one outstanding issue that i just can't crack is that the recogniser really struggles with "four" and "ford", which makes things a little tricky :(

Andreas Zetterlund
16-11-2020, 10:29 AM
When would you ever use the word Ford in the pacenotes?

ctbr
16-11-2020, 10:40 AM
Ford = Vadställe in Swedish
Maybe change to water / water crossing /water splash

Andreas Zetterlund
16-11-2020, 10:47 AM
Oh it's an actual word and not only a car manufacturer! My bad sorry.
Yes I would recommend water splash/water crossing since I never heard of Fords. But for the water crossings in rbr I have used bump/dip so far since I don't think the water itself does anything right? More the shape of the road that unsettles the car.

mr_belowski
16-11-2020, 11:03 AM
I'm not sure about this. Personally, I use 'ford' - stages like the Richard Burns Rally 'rally school' stage have multiple water crossings and it's natural (for me) to use the word "ford". I can add alternates to this so "water splash", "ford" and "water crossing" all mean the same thing but I'm reluctant to remove "ford".

I can hack around it in the code so if the parser completes its work and it's been unable to find a description for a corner direction, and the recognised text contains "right ford" / "left ford" or "ford right" / "ford left", then i can replace "ford" with "four" and re-run the parser. But this is very English specific and really nasty

Andreas Zetterlund
16-11-2020, 12:13 PM
Would it maybe be possible to add co-driver specific options so that you can for example choose which words you want for water crossing? In general I have never had my "four" being replaced by "Ford" so I don't mind it staying in as default.

Another note that I remember being used is "down" and "up". For example "100 down right 2" would mean that you are going 100 meters down a hill and probably need to brake earlier, same thing applies for "up" but I never actually used up since it usually works to just say "100 right 2 over crest" when it's a right 2 on top of a hill. But using down can be very helpful for example in places like Monte Carlo rallies where it's many downhill sections into hairpins where you need to brake WAY earlier into hairpins then if it's flat. I don't know the stage but there are some stages in the alps where it's "left 4 over crest 100 hairpin right" and you need to break before the "left 4" since it's a very steep hill so I would like to change that note to "left 4 over crest, down 100 hairpin right".

Maybe this is getting to a point where it's to much of my own personal notes but I would love to hear if it's used by others. I know down is being used plenty in rallies where the organizers hand out the notes and we don't do recce.

mr_belowski
16-11-2020, 12:38 PM
the speech recogniser is configurable - there's information about how to do this in the app's default version of the speech recogniser config, which is here:

C:\Program Files (x86)\Britton IT Ltd\CrewChiefV4\speech_recognition_config.txt. The top of this file describes the process, it's pretty simple.

there's already "up hill" and "down hill" obstacle types, which might achieve what you want?

Andreas Zetterlund
19-11-2020, 07:39 AM
I have found a bug with the latest version of CC. Correction while in stage mode does not work any longer, it does not find which turn I want to be correction, in recce mode it works.

mr_belowski
19-11-2020, 08:17 AM
yes, I can see what the problem is here and i'll release a fix for it soon (or maybe a beta, depending on whether I can get time to test properly). While I can fix the main issue, I think the correction code needs to be reworked to fit better with the recent changes to make the recce parser more flexible

Andreas Zetterlund
20-11-2020, 07:23 AM
Found another "bug", or maybe it was intended. But when I use plus and minus in my calls it calls it before the note. For example I want a corner to be "Right 2 plus" and it would call it as "plus right 2". Is it possible to make the notes 1:1 so that it will always call it in the order that I'm saying then maybe it's more work for me but I will always get the notes as I like it.

mr_belowski
20-11-2020, 07:45 AM
So you get the plus before the corner even if your pace note only contained 1 corner? Or is this happening when you have a sequence of notes in a single voice command?

mr_belowski
20-11-2020, 09:02 AM
answering my own question here...

I can repro this issue but only when i have more than one corner or obstacle in a single pace note command. "Right three then left four tightens" for example, doesn't work correctly - the "tightens" modifier is applied to the "Right three" note, which is clearly wrong.

This will be fixed in the next release (i've just fixed the code now, need to test some more). The next release will organise the modifiers properly so they apply to the corner / obstacle immediately before them in the command

Andreas Zetterlund
20-11-2020, 09:57 AM
Sorry I didn't have time to test properly if it was a sequence of corners or normal corners. But great you had time to look at it!
Is it possible to have the option to remove to between two corners? For example "Right 5 tightens to 2", I would rather have it as "Right 5 tightens 2". Just keeping the talking to a minium to get the most information possible in the fastest possible time.

Fast worker, already fixed :D
Do you think the fix will be released before 23rd? Doing a rally that ends then and would love to use your co-driver for this rally again.
Also will the update contain the correction fix?

Thanks!

mr_belowski
20-11-2020, 10:07 AM
the "...to..." is part of the sound file. I won't be able to get that sorted before the next release but can add it to the list of stuff to look at

Expect a release in the next day or so with these changes

The basic fix for the broken correction logic will be in this release, it'll take (much) longer to rework the correction stuff to make it fit better with the more flexible parser and other changes

Andreas Zetterlund
21-11-2020, 09:03 PM
So today I found out that doing "correction: earlier" will remove the pacenote. That explains why I sometimes just miss an entire note. This is during recce, I have not tested yet in race mode if it removes it when doing correction earlier, I'm sitting here trying to edit the .json file by hand hehe. Good thing CC stores the entire log with what it removed and at what distance.

mr_belowski
22-11-2020, 10:03 AM
I've been able to reproduce that. There are 2 issues - "earlier" / "later" were not implemented for recce mode corrections, and if the parser didn't find any recognisable commands in a recce mode correction call the app was removing the most recently added calls. This was a regression.

Both are fixed for the next release. In recce mode, saying "correction, later" will move all the calls in the last batch (the calls recorded in the last command) so their trigger points are 50 metres later in the stage (earlier moves them 50 metres back). Saying "correction, right three earlier" will replace the last batch with "right three" and move the trigger point of that so it's 50 metres earlier than the last batch's trigger point

mr_belowski
28-11-2020, 10:27 AM
Hi Andreas, here's a beta with those changes. The 'correction' stuff should work better in recce mode and it won't delete the last message if the correct call contains only 'later' or 'earlier', of if the correction call doesn't get parsed properly.

We'll probably release this some time in the next few days, but if you need these changes now here they are:

http://crewchief.isnais.de/CrewChiefV4-4.15.0.4_beta.msi

Andreas Zetterlund
28-11-2020, 12:38 PM
Thank you so much for taking your time. Ya I haven't dare to do a recce in the last few days after I noticed the bug since if I stutter or I forget what to say the entire recce would be ruined almost because of notes missing. And some stages that are 10+ km, takes more than 20 min to recce hehe.
Great work!

mr_belowski
28-11-2020, 01:53 PM
bare in mind that each time you do a recce session it renames the previous recce session's json file so even when it makes a mess you should be able to get back to previous notes

Andreas Zetterlund
12-12-2020, 10:11 AM
Hi again! How's it going? Have you had any time to look at "to" instead of just tightens 2?
Also is it possible to add short & half long? as length modifiers

I noticed today also that the parser handles even 3 corners in 1 phrase. I did a "Left 5 into right 4 into 1" and it read it as "Left 5 into right 4, left 1".
Not perfect but it read all the notes

mr_belowski
12-12-2020, 04:40 PM
Been snowed under with normal work stuff and getting the AMS VR implementation as good as it can be, so i've not looked at any more rally stuff.

I think the speech recogniser is close to its limit already with the phrases we currently have. I'm not sure if adding a general "to" word to the phrases is sensible but i'll have a look when i get some time. Short and half-long might be a bit safer

Andreas Zetterlund
12-12-2020, 04:54 PM
I'm sorry I phrased myself very wrong in the last post! I was meaning the removal of "to". Right not it says for example "Right 5 tightens to 2" instead of "right 5 tightens 2". So the removal of to was my question, sorry.
Ya I've been keeping an eye on the AMS VR uploads, can't wait to give it a try in the trophy trucks, getting dizzy right away.

mr_belowski
12-12-2020, 05:17 PM
Take a look at the speech recognition config file in the app install folder. There are customisation instructions at the top. It should be simple enough to change tightens to X so it's just tightens X, or have both options

[edit]

I was about to add this, but it's already there:



RALLY_TIGHTENS_TO_1 = tightens to one:tightens one:tightens acute:tightens to acute
RALLY_TIGHTENS_TO_2 = tightens to two:tightens two:tightens kay:tightens to kay
RALLY_TIGHTENS_TO_3 = tightens to three:tightens three:tightens slow:tightens to slow
RALLY_TIGHTENS_TO_4 = tightens to four:tightens four:tightens medium:tightens to medium
RALLY_TIGHTENS_TO_5 = tightens to five:tightens five:tightens fast:tightens to fast
RALLY_TIGHTENS_TO_HAIRPIN = tightens to hairpin:tightens hairpin

Andreas Zetterlund
16-12-2020, 04:48 PM
Noticed a bug with my corrections, it sometimes add a new note into the system. I've had it in a few examples where it said the distance twice after i did correct earlier.

For example here is from the corrections.json file.


{
"Distance": 776.8123,
"CorrectedDistance": 726.8123,
"Pacenote": "detail_tightens_to_2",
"Modifier": "none",
"RawVoiceCommand": "correction earlier"
},

The real note is a left 3 tighens 2 but and it calls out the note "left 3 tightens 2, 200" and then says "tightens 2" right after that again. I think something funny is happening with the parser and correction earlier hehe

Andreas Zetterlund
10-01-2021, 11:10 AM
Where does CC read notes from when there is no notes made? Is it possible to add notes from others pacenote plugin and make CC read them?

Also is it just me that has problem with 4s and 5s? CC always thinks my 5 is a 4 the first time I say it untill I correct it. It always understands when I say "correction left 5" but not when I say "left 5" then it think it's a 4. Even tho I'm saying it the exact same way

FabzGrA
24-01-2021, 05:18 PM
Sorry, I know it's not AC.
Is there any way to use the reccy function in Assetto Corsa? Pacenotes are working but I would love to create some pacenotes, spoken by Jim, for normal tracks, like Nordschleife.
I already tried the following:
-assign talk button
- say "start stage reccy" (he doesn't understand me)

Different approach:
- used CC to create own pacenotes
- copied a few sound files from the codriver folder
- pasted into the pacenotes folder
- exchanged filenames in the pacenotes.json

This works when "playing pacenotes" is active but it's really time-consuming to edit the .JSON like that.

mr_belowski
25-01-2021, 09:53 AM
the stage notes stuff loads a completely different set of commands into the speech recogniser. Allowing stage notes features for circuit racing games would mean loading the circuit racing voice commands as well as the rally voice commands - I think this would be far too many for the speech recogniser to work effectively.

This doesn't mean what you're suggesting is impossible, but it's more work than it might appear. I'd have to provide some mechanism for allowing rally-mode to be forced for circuit racing games to allow the stage notes to be created. And even then, is this actually useful for regular tracks? I appreciate that Nords is a special case here but in terms of the amount of work, it's probably less effort for me to finish the Nords pace notes I started creating (I got about half way through)

FabzGrA
25-01-2021, 10:50 AM
the stage notes stuff loads a completely different set of commands into the speech recogniser. Allowing stage notes features for circuit racing games would mean loading the circuit racing voice commands as well as the rally voice commands - I think this would be far too many for the speech recogniser to work effectively.

This doesn't mean what you're suggesting is impossible, but it's more work than it might appear. I'd have to provide some mechanism for allowing rally-mode to be forced for circuit racing games to allow the stage notes to be created. And even then, is this actually useful for regular tracks? I appreciate that Nords is a special case here but in terms of the amount of work, it's probably less effort for me to finish the Nords pace notes I started creating (I got about half way through)

Thank you for your fast reply! I already expected it to be harder than just "copying" a few lines. Well, I will stick to my above mentioned method:

- Record Pacenotes manually - let CC create the metadata.json to have the right spots
- Backup metadata.json
- Copy orig. soundfiles from %Appdata% to Documents/Pacenotes
- Rename accordingly, one by one as they are just numbered.
- Cruise along the track slowly to listen to your own pacenotes and at the same time, with the metadata.json open, adjust timing and insert the filenames of the new soundfiles that you want to be played
- i.e: caution.wav before a tight turn or whatever...

Mostly doing this because I got a few pals who, when hanging out together, love to drive, just don't know the tracks that well. Other than that, I think it's useful for people who are quite new to certain, more complex circuits, like i.e Nordschleife.

I will append this on my very same question in the AC support section.

ctbr
13-03-2021, 12:35 PM
Can you change Jims Number first note so that 1 > hairpin?
I would like the same notes as Janne but with Jims voice