crypto: sig - Fix oops on KEYCTL_PKEY_QUERY for RSA keys

Commit a2471684 ("crypto: ecdsa - Move X9.62 signature size
calculation into template") introduced ->max_size() and ->digest_size()
callbacks to struct sig_alg.  They return an algorithm's maximum
signature size and digest size, respectively.

For algorithms which lack these callbacks, crypto_register_sig() was
amended to use the ->key_size() callback instead.

However the commit neglected to also amend sig_register_instance().
As a result, the ->max_size() and ->digest_size() callbacks remain NULL
pointers if instances do not define them.  A KEYCTL_PKEY_QUERY system
call results in an oops for such instances:

  BUG: kernel NULL pointer dereference, address: 0000000000000000
  Call Trace:
  software_key_query+0x169/0x370
  query_asymmetric_key+0x67/0x90
  keyctl_pkey_query+0x86/0x120
  __do_sys_keyctl+0x428/0x480
  do_syscall_64+0x4b/0x110

The only instances affected by this are "pkcs1(rsa, ...)".

Fix by moving the callback checks from crypto_register_sig() to
sig_prepare_alg(), which is also invoked by sig_register_instance().
Change the return type of sig_prepare_alg() from void to int to be able
to return errors.  This matches other algorithm types, see e.g.
aead_prepare_alg() or ahash_prepare_alg().

Fixes: a2471684 ("crypto: ecdsa - Move X9.62 signature size calculation into template")
Signed-off-by: Lukas Wunner <lukas@wunner.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>