A.V.
New Member
- Joined
- Feb 16, 2009
- Messages
- 6,503
- Likes
- 1,159
Formulating Rational Field Trials and Evaluation Plan | Indian Defence Review
India’s defence procurement procedure is based on ’single-stage two-bid’ system. It entails submission of technical and commercial proposals by the invited vendors at the outset, albeit in two separate sealed covers. Technical proposals are opened initially whereas commercial proposals are kept sealed in safe custody. After technical evaluation, vendors whose products are found to be fully compliant with the laid down parameters are short listed and are considered acceptable for introduction into service. Commercial proposals of only such technically acceptable vendors are opened to identify the lowest bidder for the award of contract. In other words, technical appraisal precedes commercial assessment.
The process of technical appraisal starts with the constitution of Technical Evaluation Committee (TEC) under the concerned Service Headquarters (SHQ). It carries out paper evaluation of all technical proposals and selects vendors who should be called for field trials. Thereafter, SHQ carries out field trials and evaluation (FT&E) under its aegis.
Field trials and evaluation are two distinct activities. Field trials are conducted to validate performance parameters in actual terrain/climatic conditions and to collect and collate data as regards functional and operational capability of systems, sub-systems and components as regards their mission accomplishment potential. Evaluation is the process of review and analysis of qualitative or quantitative data produced during field trials and other tests to determine whether a system satisfies its acceptance criteria and should be accepted for induction into service or not.
In other words, evaluation flows from field trials and its quality is dependent on the exhaustiveness and credibility of the data generated during field trials. Therefore, field trials are central to the complete process of technical appraisal.
FT&E Given Inadequate Attention in India
SHQ have failed to appreciate the criticality of FT&E. This neglect is one of the primary reasons for India’s inability to conclude contracts in multi-vendor competitive environment.
Indian FT&E regime suffers from a total lack of advance planning. No comprehensive FT&E plan is evolved in advance. It is the primary reason for most of the infirmities that plague technical appraisal process. The sole objective of field trials is to generate data to enable evaluating staff to take well-considered decisions. Yet, evaluating authority is never considered prior to the issuance of trial directive. Such disjointed functioning results in the generation of incomplete and insufficient data, thereby forcing evaluating authority to either demand supplementary trials or take decisions based on assumptions/conjectures.
Trial directives are prepared in a routine manner by user directorates without adequate application of mind. The total thrust is towards listing out of Services Qualitative Requirements (SQR) and asking trial units to check them out. Many aspects remain ambiguously defined, lending themselves to multiple interpretations and subsequent misunderstandings and squabbling. Recently, field trials of light helicopters for the army were handled in such a cavalier manner that one of the vendors felt aggrieved and appealed to the Government. The process had to be aborted.
Trial units are detailed solely on the basis of their location. No attention is paid to the suitability of officers and availability of the necessary infra-structure. Worse, the detailed units consider trials to be an irksome bother and treat them in a perfunctory manner.
There is no standard format of trial reports. These are prepared as per the understanding of the trial units. Most reports contain no specific recommendations and are couched in generalities. There is a tendency on the part of trial units and the intermediate commanders to introduce new elements, thereby vitiating the report.
Evolution of Cogent FT&E Plan
The sole aim of the entire technical evaluation process is to identify equipment that is considered technically acceptable for introduction into service. To enable general staff to take well-considered decisions during the evaluation process, all required inputs should be made available to them. There should be no gaps at all. Therefore, it is essential that FT&E plan is evolved in close consultations with evaluating general staff. The aim should be to ensure that answers are obtained to all critical issues as likely to be sought by decision making authorities at different stages of the acquisition process.
Apparently, preparation of a cogent FT&E plan requires thorough knowledge of intended exploitation of equipment and assignment of missions. The process of evolving a detailed FT&E plan must commence concurrently with the issuance of Request for Proposals (RFP). Once underlying philosophy and broad policy are finalised, an outline plan of action should be drawn to achieve the following objectives:
(a) Validation of SQR as mentioned in RFP -India’ defence procurement procedure (DPP) mandates that equipment complying with all SQR can only be considered technically acceptable and no deviation is permissible in this regard.
(b) Generation of additional data to facilitate holistic evaluation and decision making - regular interaction with evaluating authority is essential to ensure that all required aspects get covered and that there are no critical gaps in inputs.
By the time TEC shortlists vendors who should be called for trials, FT&E plan should be fully ready along with the trial directive. Some of the major aspects that need attention are discussed subsequently.
Amplification of SQR
Although DPP stipulates that SQR should lay down user’s requirements in a comprehensive, structured, concrete and verifiable manner, it has been seen that they are generally formulated in imprecise and broad terms. SQR like ‘provision of adequate comfort to a driver’ or ‘robust cross-country performance for prolonged duration’ lend themselves to multiple interpretations. Such SQR when issued to trial units as part of the trial directive create further confusion. It is extremely difficult to determine and evaluate ‘driver’s comfort’ of competing vehicles, as it is an individual-specific and indeterminate quality. Similarly, ‘cross-country robustness’ and ‘prolonged duration’ mean differently to different people.
FT&E plan should contain no parameter which is not spelt out in specific and verifiable terms. Therefore, it is essential that all SQR are converted into unambiguous and precise questions which can be answered by the trial units without any misunderstanding.
Phasing of Trial
Procurements are carried out on the basis of SQR. SQR are minimum military requirements, given in terms of functional characteristics, which are essential to the task/tasks to be performed by the equipment being procured. Therefore, all SQR are vital and must be complied with. However, there are invariably one or more SQR that are considered central to mission accomplishment capability and hence are required to be accorded primacy. For example, bullet proof vehicles are procured primarily for protection against designated hostile fire. Therefore, all other SQR become secondary to bullet proofing.
Such fundamental characteristics without whose compliance procurement becomes meaningless are termed as vital performance attributes (VPA). An attribute is an assessable characteristic that describes an aspect of a system or capability. In the above mentioned case, it is essential to first ascertain that the vehicle satisfies VPA of protection before undertaking validation of other SQR. Howsoever good a vehicle may be it cannot be considered for acceptance if it fails to provide required degree of protection. Similarly, ceiling and load become VPA for helicopters being procured for support to high altitude operations.
Field trials are extremely expensive and consume considerable resources and time. By restricting detailed trials to VPA-compliant equipment, considerable infructuous effort and time can be saved. For that, field trials may be required to be held in two phases. In Phase 1, all competing equipments are tested for VPA compliance. Thereafter, only successful equipments are subjected to detailed trials in Phase 2. FT&E plan should, therefore, be based on identification of VPA and required phasing of trials.
Trial Methodology
After listing all SQR, the best mode of their validation should be determined. Whereas most of the tests can be carried out in field conditions by the nominated trial units, many parameters require controlled testing environment (temperature, humidity and so on). It is incorrect to ask a field trial unit whether the colour of a fabric bleeds when it comes in contact with water, kerosene and oil. Such tests need controlled conditions and can best be done in establishments and laboratories having necessary infra-structure. Likewise, destruction tests requiring use of explosives/ammunition need safeguarded facilities with proof testing instrumentation.
Although trials should be carried out in terrain and climatic conditions in which the use of equipment is envisaged during operations, there may be cases where field trials are not feasible in India due to lack of necessary testing facilities. FT&E plan should examine whether modelling and computer simulation would meet the requirement or trial teams should visit vendor’s premises abroad.
In cases where more than one service is procuring the item, close coordination is required to ensure that operational needs of all the buyer services are met. Therefore, the lead service has to involve other services in the formulation of FT&E plan. There should neither be any duplication of effort nor omission of any essential aspect. In case the equipment under trial has interfaces with equipment held by another service, due assistance and advice should be sought from it.
In case of urgently required items, it may become necessary to hasten trial process. While formulating trial methodology, feasibility of carrying out concurrent trials should be explored. It has been seen that by prudent scheduling of trial activities, a number of performance and maintainability tests can be held simultaneously and trial process telescoped. Therefore, validation of the support system and maintainability trials, integral to and complementing the trial programme of the weapon system should be held simultaneously, wherever feasible.
Need for Ascertaining Operational Exploitability
As no service equipment is ever used in isolation, its true potential can only be gauged when exploitation trials are carried out in conjunction with associated components of the services. Besides validating performance characteristics of equipment, potential for mission accomplishment in consonance with overall objectives is also assessed. At times, full fledged field exercises may be required to be undertaken to establish conclusive results. Such trials will be holistic in nature rather than single equipment centric.
Operational exploitation tests focus on operational requirements, effectiveness and suitability, and not solely on compliance of technical performance parameters. Therefore, FT&E plan must factor in the above requirement to ascertain suitability of equipment as regards suitability of current tactics and organisational structure. The plan must also examine requirement to validate operating procedures, training and maintenance support of the equipment.
Selection of Trial Units
Trial of new equipment with newly developed and diverse technologies is a highly specialised task. Therefore, it is essential that the selection of trial units is carried out after considerable deliberation. The following aspects need attention:-
(a) Does the unit possess adequate competencies to be able to validate the specified performance attributes in a systematic, logical and comprehensive manner?
(b) Does the unit have the necessary expertise and vision to be able to interpolate exploitation of equipment in diverse combat milieu?
(c) Does the unit possess spare capacity to undertake trials?
(d) Does the unit possess or have access to necessary infrastructure?
(e) Does the unit need additional expertise by co-opting of specialists from DRDO and/or help from training organisations?
(f) Type and nature of briefing required to be given to the units.
Format of Trial Reports
Trial reports submitted by different trial units form the basis of subsequent staff evaluation. Poorly compiled reports result in protracted correspondence and repeated queries. More importantly, such reports tend to lose credibility as regards assiduousness and objectivity. At times additional trials may have to be ordered to obtain specific inputs to cover gaps, resulting in delays and avoidable expenditure.
FT&E plan should include format of trial report which the trial units would be required to fill. They should be asked to stick to the issues under validation and avoid introducing extraneous factors, thereby forcing trial units to give their replies and recommendations in specific and unambiguous terms. Similarly, intermediate commanders should be asked to comment only on the suitability of equipment for induction into the service and not introduce their own viewpoints.
The Way Forward
As seen above, India has to pay more attention to FT&E as it is by far the most critical activity in the entire procurement process. It has to be appreciated that FT&E has to be equipment specific and there cannot be routine application of a standard format. Every procurement proposal is unique and has to be treated differently. Some of the steps that should be taken are as follows:-
DPP mandates that trial methodology must be indicated in RFP. It implies that broad contours of FT&E plan should be finalised before the issuance of RFP. Therefore, the process of formulation of FT&E plan must be initiated while seeking sanction for the procurement proposal.
FT&E plan should be evolved in an integrated manner. The current method of compartmentalised functioning must be replaced by integrated and collegiate decision making. All stake holders must be taken on board to ensure that the finalised plan meets all requirements and no critical gaps appear later on.
Criteria for technical evaluation must be unambiguous and assessable, both qualitatively and quantitatively. The plan must be rational and all-inclusive, covering underlying philosophy, policy and roadmap of the whole process. It should also contain format of trial reports to ensure meticulous treatment of all required aspects.
Quality of FT&E is totally dependent on the diligence and expertise of the functionaries involved. For example, determination of sequence of trials requires a thorough knowledge of primary tasks for which equipment is being procured. Therefore, selection of concerned staff requires careful consideration. Additionally, they should be given specialised training and equipped with necessary wherewithal.
The paramount objective of technical evaluation is to ascertain whether the equipment fulfills the mission for which it is being procured. FT&E entails deliberate and rational generation of performance data and its conversion into constructive inputs for the decision maker. For that, test and evaluation process has to be so structured that all information required for assessing attainment of technical performance parameters and operational effectiveness of equipment is made available to the evaluating authority. Finally, a well formulated FT&E plan ensures that focus on essentials is not lost during the whole process by eliminating infructuous digressions.
Maj Gen Mrinal Suman
India’s defence procurement procedure is based on ’single-stage two-bid’ system. It entails submission of technical and commercial proposals by the invited vendors at the outset, albeit in two separate sealed covers. Technical proposals are opened initially whereas commercial proposals are kept sealed in safe custody. After technical evaluation, vendors whose products are found to be fully compliant with the laid down parameters are short listed and are considered acceptable for introduction into service. Commercial proposals of only such technically acceptable vendors are opened to identify the lowest bidder for the award of contract. In other words, technical appraisal precedes commercial assessment.
The process of technical appraisal starts with the constitution of Technical Evaluation Committee (TEC) under the concerned Service Headquarters (SHQ). It carries out paper evaluation of all technical proposals and selects vendors who should be called for field trials. Thereafter, SHQ carries out field trials and evaluation (FT&E) under its aegis.
Field trials and evaluation are two distinct activities. Field trials are conducted to validate performance parameters in actual terrain/climatic conditions and to collect and collate data as regards functional and operational capability of systems, sub-systems and components as regards their mission accomplishment potential. Evaluation is the process of review and analysis of qualitative or quantitative data produced during field trials and other tests to determine whether a system satisfies its acceptance criteria and should be accepted for induction into service or not.
In other words, evaluation flows from field trials and its quality is dependent on the exhaustiveness and credibility of the data generated during field trials. Therefore, field trials are central to the complete process of technical appraisal.
FT&E Given Inadequate Attention in India
SHQ have failed to appreciate the criticality of FT&E. This neglect is one of the primary reasons for India’s inability to conclude contracts in multi-vendor competitive environment.
Indian FT&E regime suffers from a total lack of advance planning. No comprehensive FT&E plan is evolved in advance. It is the primary reason for most of the infirmities that plague technical appraisal process. The sole objective of field trials is to generate data to enable evaluating staff to take well-considered decisions. Yet, evaluating authority is never considered prior to the issuance of trial directive. Such disjointed functioning results in the generation of incomplete and insufficient data, thereby forcing evaluating authority to either demand supplementary trials or take decisions based on assumptions/conjectures.
Trial directives are prepared in a routine manner by user directorates without adequate application of mind. The total thrust is towards listing out of Services Qualitative Requirements (SQR) and asking trial units to check them out. Many aspects remain ambiguously defined, lending themselves to multiple interpretations and subsequent misunderstandings and squabbling. Recently, field trials of light helicopters for the army were handled in such a cavalier manner that one of the vendors felt aggrieved and appealed to the Government. The process had to be aborted.
Trial units are detailed solely on the basis of their location. No attention is paid to the suitability of officers and availability of the necessary infra-structure. Worse, the detailed units consider trials to be an irksome bother and treat them in a perfunctory manner.
There is no standard format of trial reports. These are prepared as per the understanding of the trial units. Most reports contain no specific recommendations and are couched in generalities. There is a tendency on the part of trial units and the intermediate commanders to introduce new elements, thereby vitiating the report.
Evolution of Cogent FT&E Plan
The sole aim of the entire technical evaluation process is to identify equipment that is considered technically acceptable for introduction into service. To enable general staff to take well-considered decisions during the evaluation process, all required inputs should be made available to them. There should be no gaps at all. Therefore, it is essential that FT&E plan is evolved in close consultations with evaluating general staff. The aim should be to ensure that answers are obtained to all critical issues as likely to be sought by decision making authorities at different stages of the acquisition process.
Apparently, preparation of a cogent FT&E plan requires thorough knowledge of intended exploitation of equipment and assignment of missions. The process of evolving a detailed FT&E plan must commence concurrently with the issuance of Request for Proposals (RFP). Once underlying philosophy and broad policy are finalised, an outline plan of action should be drawn to achieve the following objectives:
(a) Validation of SQR as mentioned in RFP -India’ defence procurement procedure (DPP) mandates that equipment complying with all SQR can only be considered technically acceptable and no deviation is permissible in this regard.
(b) Generation of additional data to facilitate holistic evaluation and decision making - regular interaction with evaluating authority is essential to ensure that all required aspects get covered and that there are no critical gaps in inputs.
By the time TEC shortlists vendors who should be called for trials, FT&E plan should be fully ready along with the trial directive. Some of the major aspects that need attention are discussed subsequently.
Amplification of SQR
Although DPP stipulates that SQR should lay down user’s requirements in a comprehensive, structured, concrete and verifiable manner, it has been seen that they are generally formulated in imprecise and broad terms. SQR like ‘provision of adequate comfort to a driver’ or ‘robust cross-country performance for prolonged duration’ lend themselves to multiple interpretations. Such SQR when issued to trial units as part of the trial directive create further confusion. It is extremely difficult to determine and evaluate ‘driver’s comfort’ of competing vehicles, as it is an individual-specific and indeterminate quality. Similarly, ‘cross-country robustness’ and ‘prolonged duration’ mean differently to different people.
FT&E plan should contain no parameter which is not spelt out in specific and verifiable terms. Therefore, it is essential that all SQR are converted into unambiguous and precise questions which can be answered by the trial units without any misunderstanding.
Phasing of Trial
Procurements are carried out on the basis of SQR. SQR are minimum military requirements, given in terms of functional characteristics, which are essential to the task/tasks to be performed by the equipment being procured. Therefore, all SQR are vital and must be complied with. However, there are invariably one or more SQR that are considered central to mission accomplishment capability and hence are required to be accorded primacy. For example, bullet proof vehicles are procured primarily for protection against designated hostile fire. Therefore, all other SQR become secondary to bullet proofing.
Such fundamental characteristics without whose compliance procurement becomes meaningless are termed as vital performance attributes (VPA). An attribute is an assessable characteristic that describes an aspect of a system or capability. In the above mentioned case, it is essential to first ascertain that the vehicle satisfies VPA of protection before undertaking validation of other SQR. Howsoever good a vehicle may be it cannot be considered for acceptance if it fails to provide required degree of protection. Similarly, ceiling and load become VPA for helicopters being procured for support to high altitude operations.
Field trials are extremely expensive and consume considerable resources and time. By restricting detailed trials to VPA-compliant equipment, considerable infructuous effort and time can be saved. For that, field trials may be required to be held in two phases. In Phase 1, all competing equipments are tested for VPA compliance. Thereafter, only successful equipments are subjected to detailed trials in Phase 2. FT&E plan should, therefore, be based on identification of VPA and required phasing of trials.
Trial Methodology
After listing all SQR, the best mode of their validation should be determined. Whereas most of the tests can be carried out in field conditions by the nominated trial units, many parameters require controlled testing environment (temperature, humidity and so on). It is incorrect to ask a field trial unit whether the colour of a fabric bleeds when it comes in contact with water, kerosene and oil. Such tests need controlled conditions and can best be done in establishments and laboratories having necessary infra-structure. Likewise, destruction tests requiring use of explosives/ammunition need safeguarded facilities with proof testing instrumentation.
Although trials should be carried out in terrain and climatic conditions in which the use of equipment is envisaged during operations, there may be cases where field trials are not feasible in India due to lack of necessary testing facilities. FT&E plan should examine whether modelling and computer simulation would meet the requirement or trial teams should visit vendor’s premises abroad.
In cases where more than one service is procuring the item, close coordination is required to ensure that operational needs of all the buyer services are met. Therefore, the lead service has to involve other services in the formulation of FT&E plan. There should neither be any duplication of effort nor omission of any essential aspect. In case the equipment under trial has interfaces with equipment held by another service, due assistance and advice should be sought from it.
In case of urgently required items, it may become necessary to hasten trial process. While formulating trial methodology, feasibility of carrying out concurrent trials should be explored. It has been seen that by prudent scheduling of trial activities, a number of performance and maintainability tests can be held simultaneously and trial process telescoped. Therefore, validation of the support system and maintainability trials, integral to and complementing the trial programme of the weapon system should be held simultaneously, wherever feasible.
Need for Ascertaining Operational Exploitability
As no service equipment is ever used in isolation, its true potential can only be gauged when exploitation trials are carried out in conjunction with associated components of the services. Besides validating performance characteristics of equipment, potential for mission accomplishment in consonance with overall objectives is also assessed. At times, full fledged field exercises may be required to be undertaken to establish conclusive results. Such trials will be holistic in nature rather than single equipment centric.
Operational exploitation tests focus on operational requirements, effectiveness and suitability, and not solely on compliance of technical performance parameters. Therefore, FT&E plan must factor in the above requirement to ascertain suitability of equipment as regards suitability of current tactics and organisational structure. The plan must also examine requirement to validate operating procedures, training and maintenance support of the equipment.
Selection of Trial Units
Trial of new equipment with newly developed and diverse technologies is a highly specialised task. Therefore, it is essential that the selection of trial units is carried out after considerable deliberation. The following aspects need attention:-
(a) Does the unit possess adequate competencies to be able to validate the specified performance attributes in a systematic, logical and comprehensive manner?
(b) Does the unit have the necessary expertise and vision to be able to interpolate exploitation of equipment in diverse combat milieu?
(c) Does the unit possess spare capacity to undertake trials?
(d) Does the unit possess or have access to necessary infrastructure?
(e) Does the unit need additional expertise by co-opting of specialists from DRDO and/or help from training organisations?
(f) Type and nature of briefing required to be given to the units.
Format of Trial Reports
Trial reports submitted by different trial units form the basis of subsequent staff evaluation. Poorly compiled reports result in protracted correspondence and repeated queries. More importantly, such reports tend to lose credibility as regards assiduousness and objectivity. At times additional trials may have to be ordered to obtain specific inputs to cover gaps, resulting in delays and avoidable expenditure.
FT&E plan should include format of trial report which the trial units would be required to fill. They should be asked to stick to the issues under validation and avoid introducing extraneous factors, thereby forcing trial units to give their replies and recommendations in specific and unambiguous terms. Similarly, intermediate commanders should be asked to comment only on the suitability of equipment for induction into the service and not introduce their own viewpoints.
The Way Forward
As seen above, India has to pay more attention to FT&E as it is by far the most critical activity in the entire procurement process. It has to be appreciated that FT&E has to be equipment specific and there cannot be routine application of a standard format. Every procurement proposal is unique and has to be treated differently. Some of the steps that should be taken are as follows:-
DPP mandates that trial methodology must be indicated in RFP. It implies that broad contours of FT&E plan should be finalised before the issuance of RFP. Therefore, the process of formulation of FT&E plan must be initiated while seeking sanction for the procurement proposal.
FT&E plan should be evolved in an integrated manner. The current method of compartmentalised functioning must be replaced by integrated and collegiate decision making. All stake holders must be taken on board to ensure that the finalised plan meets all requirements and no critical gaps appear later on.
Criteria for technical evaluation must be unambiguous and assessable, both qualitatively and quantitatively. The plan must be rational and all-inclusive, covering underlying philosophy, policy and roadmap of the whole process. It should also contain format of trial reports to ensure meticulous treatment of all required aspects.
Quality of FT&E is totally dependent on the diligence and expertise of the functionaries involved. For example, determination of sequence of trials requires a thorough knowledge of primary tasks for which equipment is being procured. Therefore, selection of concerned staff requires careful consideration. Additionally, they should be given specialised training and equipped with necessary wherewithal.
The paramount objective of technical evaluation is to ascertain whether the equipment fulfills the mission for which it is being procured. FT&E entails deliberate and rational generation of performance data and its conversion into constructive inputs for the decision maker. For that, test and evaluation process has to be so structured that all information required for assessing attainment of technical performance parameters and operational effectiveness of equipment is made available to the evaluating authority. Finally, a well formulated FT&E plan ensures that focus on essentials is not lost during the whole process by eliminating infructuous digressions.
Maj Gen Mrinal Suman
