Hello Pramod,
Nobody can answer your questions correctly unless he knows much more about your system.
All I can do to suggest few things...
It is large project, with code banking >> 2 banks around 100 K, XDATA > 64 K, RTC, Serial Interface RS232 etc.
Implemented own / sourced written RTOS, It is having upto 64 similer tasks, means all the tasking working on same type of work depending on requirements.
>> lot of peripherals IC's, like, SRAM, RTC, DTMF Receiver, Generator, Latch, Logics, LED etc..
Start small, man. No one will be able to do all that in one go

Since you have implemented your own RTOS, you must check it throughly. It's daunting task to write a RTOS, better to use a commercial one (atleast we can shout at someone, when something goes wrong

)
system is working but no so stable, facing problems, like memory corruption.
Check your PCB layout, is it correct ? It may not be enough just to have right connections. Look carefully at address/data lines. Is there any glitch ? Is any noisy component very near to ram ics ? If using dynamic rams, is refresh circuit working correctly ? Are you well within spec of ram ic's access timings ?
Now, run memory extensive memory dignostics code, just to be sure that all hardware is ok.
Check your software. It is easy overlook the pointers in C. They certainly ease many tasks, but can be nightmare if you try to mess with them. Is some code, espcially RTOS's system code, de-referencing a null pointer ? Better include code to check pointer for null, before accessing them.
Are you using malloc() & it's friends. Check compiler's output listing(better source if you have) for the these beasts. Carefully run some test code & see if it chokes at something !
You haven't mentioned what type RTOS you are using. If it is pre-emptive, see if all tasks are following the RTOS's method of data exchange-semaphore, queues or pipes whatever it may be.
& no task is monkeying with data when ot shouldn't. Better make most data static & write reentrant functions to access them using semaphores, so that no two tasks or task & interrupt access can access them simultaneously. Every data shared between two tasks, must be doene so atomically.
Make sure no task is calling functions which are not re-entrant, c compiler's library is typically not re-entrant. Your compiler manual must tell you about it. If it does not- chuck that thing right out of window-it's not worth it.
Finally, is stack blowing up ? Does each task has enough stack space ? Or multiple ISRs are causing stack overflow ? Does RTOS doing the stack manipulation correctly, during task switch ?
ISR is big, can't keep small as requirements, ISR time is 20 mSec. on 11.0592 MHz Crystal.
That doesn't give ISRs all specs. You must specify 1) ISR's worst case exec. time 2) ISR's frequecny 3) ISR's priority 4) Worst case exec. time of any other interrupt having priority greater than this ISR. Unless you know all this, you can't comment on how much time is available for tasks or other ISRs.
Well, I must say, you should have written ISR specs before you even begin to write your first line of code.
Demontask or Superloop is also big, lot of conditions, similar work for all the tasks.
You have 64 tasks & still your superloop is too big

You certainly have to take serious look at your code structure.
Lot of peripherals IC's, like, SRAM, RTC, DTMF Receiver, Generator, Latch, Logics, LED etc..
1. How to fix or deside ISR timing. 15 mSec is generally required to get the tone cadence.
Try using slave CPUs. Since you already have I2C bus, you can hook them on to it & transfer some of the load to these slaves. There will be great improvement is overall system performance & negligible cost.
3.How to fix the ISR time. >> presently 20 mSec for 40 task system and same for 10 task system.
4. demontask is around 50 mSec but not stable, lot of fluctuations. in CRO >> showing 30 - 40 - 60 - 35 - 55- 60 etc.
5. If ISR is 15 mSec. the demontask are some what stable >> 55 - 60 - 55- 60 - 50 etc.
6. In smaller system 10 tasks, demontask are completely unstable. The Crystal is same as 11.0592 MHz.
Nothing can be said clearly about this, unless actual code & system specs are seen.
How the tasks are prioritized ? Does ISR exec time depend on main-line code or no. of tasks ? Then ISR is coded incorrectly. Look for ways to move such code out of ISR. Make ISR exec. timings deterministic as far as possible. Unless you know how much your ISRs take time, how can you decide how much work can be done in main-line ! If your ISR is taking 20mSec to execute, then certainly, that's big no no !!
2. If we keep ISR Short, demontask become long, Watch dog Reset the system.
Again serous problems with overall code structure. You must do the entire system time-map.
3. If we keep ISR Dynamic, skip the functions >> it shows Unstable on CRO.
What you mean by Dynamic ISR ??
4. If ISR or demontask is unstable on CRO, it gives any negative results. Like memory corrputions etc.
Typical of shared data bug. That's ISR not saving/restoring task context correctly, tasks accessing shared data non-atomically or even worse interrupt is getting nested when RTOS doesn't support it !
5. Any document is available for this reference.
6. What is a basic theory between ISR & Demontask timing.
Google for real time systems, you will get many. Most does not go beyond introduction, others go too much mathematical & put you to sleep. There is no hard & fast routes to success, but there are some rules of thumb which gives satisfactory (may not be the best) results. Best way is to learn basic principles & experiment constantly.
I've some data in my backup. I'll dig it up & post as soon as i can.
Hope this shades some light.
BTW, it is nice to see that someone is still using 8051 for such big systems.
regards,
sam_des