[Tool Idea] Speech Recognition in NBA 2K13 PC using MS SDK

[Leftos]

Got you interested, didn't I? Heh. This idea actually comes from a thread over at OS, where someone asked whether we could get the advertised Kinect functionality on the PC, due to the existence of the Kinect SDK for Windows. The fact that this SDK exists means that you can buy a Kinect and use it on the PC, if programs make use of that SDK. There's not a lot of programs using it though.

However advanced Kinect's speech recognition is, Microsoft has a general Speech SDK out there, which means that with a bit of programming, we can get speech recognition tools working with any microphone.

Granted, we won't be able to build advanced functionality since we don't have access to the game's code, but using programs like autoHotKey, we could simulate key presses and combinations of them. You could for example call "TIMEOUT" into your headset, and have the tool translate that into the press of the Back button.

Is it simpler and more safe to use the Back button instead? Sure. Would it be a fun thing to try out to see if possible? GOD YES.

I'll start working on it come October hopefully, but I want to see whether people would be interested in something like this, what functionality they'd like to see in it (remember, you have to suggest something that can be done with a key/button combination), and whether they'd be willing to donate towards me getting a Kinect to try this out with.

So, what do you think?

[JaoSming]

would be interesting with coach mode enabled

[dare]

Good idea. I wish I can help you with this but my programming skills doesn't even come near yours by a very large margin. Anyways good luck man

[shorty_ubk]

I still don't know that's true but Win8'll be compatible with Xbox and we can use kinect in our PC's... So if it'll be true, maybe you can use this feature for your advantage... It'll be a lot easier for you to make a tool like that...

[Leftos]

Windows 8's Xbox Live integration and "Xbox Games for Windows" initiative have nothing to do with this.

Kinect already has a Windows SDK, so Kinect owners can plug the Kinect to their computer and use it with applications which make use of it. There's also a non Kinect-specific SDK for general Speech Recognition, that would allow me to make the tool work with any microphone, and not just Kinects.

[shorty_ubk]

Windows 8's Xbox Live integration and "Xbox Games for Windows" initiative have nothing to do with this.

Kinect already has a Windows SDK, so Kinect owners can plug the Kinect to their computer and use it with applications which make use of it. There's also a non Kinect-specific SDK for general Speech Recognition, that would allow me to make the tool work with any microphone, and not just Kinects.

I got it sir, but maybe it'll make things easier and more accurate for you, that's what I want about your mod...

[Andrew]

Very cool idea, worth a shot.

[Nick]

Does that mean we can only voice activate things that there's already a button for?

[Leftos]

We don't get to interact with the game's features in any other way, so pretty much only things we can voice-activate is a button or a combination of buttons, like a macro. For example, calling out "POST UP" and having the macro "D-Pad Right, Y" be acted out automatically and instantly, so you can focus on controlling the current player.

[Leander]

This is useless in my opinion. Many games implement such features these days just to be part of a new hype but I don't understand all the excitement at all. In Skyrim for example you can switch weapons or even use the shouts in Dragon Tongue with a microphone but what for? It might be "cool" and "awesome" for teenagers or children who are impressed by these new features but they are pretty useless for the older audience. First, speech recognition doesn't always work and you need three tries till your character finally does what you could also do in a 1/th second by a simple button press. Second, it's too complicated to actually say words and talk to a machine although you have used a gamepad for years and know all button combinations in your sleep. Third, my personal opinion is that I want to focus on gaming, I want to relax and have handy controls during some hours of playing a game. Usually, I talk a lot on the phone in my job, I talk to people in the supermarket, I talk to friends...and talking to the console for hours too just for something that can be done by simple button presses is exhausting. Apart from that, motion controls are a great invention for casual games (Wii sports etc.) but no - I don't want to run for 90 minutes when playing FIFA or jump around in front of the TV for hours.

[Leftos]

Leander, I'm not arguing about the usefulness of something like this. Anything that can be done with a button, it's safer to do with a button than hope that the computer understands what you're saying,\ and that the lag isn't terrible.

This is just a Proof of Concept I'll try to do either way, just to get some experience with the Speech/Kinect SDK. Something to add to my programming experience and CV.

I'm just starting the thread to get suggestions on what people would actually want to try out by speaking into a mic, just out of curiosity.

[Leander]

Leftos Sorry, I misread your post. I didn't read that YOU will code something so my post was just my opinion on speech recognition in games generally. I didn't mean to criticize your effort.

[Leftos]

We have a Kinect Integration Trailer, which shows what 2K is going to allow people with a Kinect to do via speech recognition, so feel free to criticize that one as much as you want.

I'm just asking what button combinations/actions people would like to have a spoken command for.

[Andrew]

While I don't have a Kinect and won't be buying one, I disagree that the concept 2K is implementing in NBA 2K13 is childish or exclusively aimed at a younger audience. It's quite innovative and potentially very useful for quickly performing certain actions, like calling for a play or lining up a substitution, without causing a break in the action. To perform those actions, particularly on-the-fly substitutions without pausing the game, you've got to forfeit control of the action temporarily while you're making the necessary input. The voice controls avoid that, at least in theory.

[Leftos]

Again, this is beyond the scope of what we can do. Those topics you bring up are all nice regarding the Kinect integration, but this project has two main differences:

1) It will hopefully use the generic Speech SDK, which will make it compatible with any PC microphone or headset.
2) Since we don't have the ability to create shortcuts in the game code or access whatever functions the Kinect does, we're limited to calling out commands and having a macro play out instantly.

I've already put up an example about this.
"Post Up!": D-Pad Right, Y
"Get Open!" D-Pad Right, X
"Timeout!": Back
"Pass to basket!": RB, LB

If you can get me more suggestions like that, no matter how complicated the macro is, maybe we can make something out of all this.

[Andrew]

Indeed, I was referring to the concept in general. In terms of what could be done here though, as you noted we could perhaps ape some of those useful features with some quick playcalling.

[shadowgrin]

Pick and roll on offense and double team on defense come to mind. Both are simple commands/button presses but important to the game irl and videogames.

[Lean]

Sounds interesting, maybe we can cook up something like this in the coming patching year?

The SDK is too sensitive with its recognition though.

[Leftos]

That was exactly my point. It's not just an idea, it's something I'm willing to work on, that's why I'm asking for ideas on what you'd like to be able to do by just saying that into your microphone.

I know every Speech Recognition kit has its problems, but it's worth a try.

[volsey]

What about:

Verbal command --> macro and action performed

Foul him --> Back Button to intentionally foul
Timeout --> Back to call timeout
Set a pick --> LB I think it had to be held ?

The different corresponding defensive plays:
Full court Press--> Right, RB, A ( or w.e it was)
Zone--> Right, w.e face button
Etc etc.

I really liked the "bring in Kobe" substitution feature from the trailer, possible workaround:
Sub-out shooting Guard or bring in SG --> Down, Circle (as circle was shooting guard)
I understand this would just default to the next SG in the rotation but most of the time that's who you want anyways.

Same could be done for the other positions/plays etc. Sorry if some of these were already suggested.

Really hope you go through with this, shouldn't be too hard if you can figure out the speech recognition sdk. If its even somewhat useful I'd throw you $5-10 for the effort.

[Leftos]

All nice ideas, however, the substitution won't work this way this year. 2K12's on the fly substitutions didn't work well, even though that system was meant to allow for quicker changes, and they're going back to the 2K11 style.

It could still be
Down, Down, Right, [insert whatever button closes the on-the-fly sub HUD here] --> switch to next SG in rotation

[shadowgrin]

If its even somewhat useful I'd throw you $5-10 for the effort.

I'm sure Leftos would find some needy children to help with that cash.

[Leftos]

Just so we're clear, all donations received as part of this project will be contributed to the "Help shadowgrin get psychological help" fund.

[shadowgrin]

I prefer you donate it to The Human Fund.

Besides, there are a lot kids that need books and that money will help greatly.

[Leftos]

Think I'll spend it all on beer.

We'll see, depends on my mood.